October 2011 Volume 14 Number 4 - Educational Technology ...

Two criteria have been adopted to evaluate the performance of waiting strategy. Reduced Precision (RP) extends the
classical Precision measure by considering the waiting time. To calculate RP, it is necessary to determine the
relevance of teaching methods to concepts. In this evaluation, the work of determining the relevance was executed by
human experts. Particularly, five professors, experts in e-Learning, were invited to judge whether a teaching method
was relevant to a Mathematical concept or not, the relevance being decided by vote.
Impact of Due Time Setting on Static Strategy
The purpose of this first experiment was to determine the importance of Due Time Setting for retrieving teaching
methods in a tutor supporting environment. A Static waiting strategy was implemented as a baseline for evaluation.
The Static strategy waits for a fixed amount of time. The participants are 4 pupils, 4 tutors and 100 supporting
teachers. Each tutor is assigned a pupil. Without loss of generality, we let the 4 pupils study the four subjects:
Numbers, Geometry, Algebra and Probabilities. Tutors use the tutor supporting environment to retrieve teaching
methods. Each pupil asks three questions about learning concepts, and the average Precision and Recall values are
derived. Figure 10 shows the results with respect to classical recall and precision. First, we observe that the Recall
value increases as Due Time increases for the four subjects. This means that an adaptive waiting strategy is
demanded and a naïve static waiting strategy will not be suitable for this scenario. Similarly, the Precision value
increases as Due Time increases for the four subjects, though not so significantly. Furthermore, we find that Due
Time Setting is suitable for the scenario of Recall-oriented teaching method retrieval, which tries to find as many
results as possible. For Precision-oriented teaching method retrieval, longer waiting might not significantly increase
the performance.
Recall
Reduced Recall
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0.04
0.035
0.03
0.025
0.02
0.015
0.01
0.005
0
Num bers
Geom etry
Algebra
Probabilities
5 10 15 20
Due Time (min)
Precision
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Numbers
Geom etry
Algebra
Probabilities
5 10 15 20
Due Time (min)
Figure 10. (a) The relation of Recall and due time; (b) the relation of Precision and due time
Numbers
Geom etry
Algebra
Probabilities
5 10 15 20
Due Time (min)
Reduced Precision
0.12
0.1
0.08
0.06
0.04
0.02
0
5 10 15 20
Due Time (min)
Num bers
Geom etry
Algebra
Probabilities
Figure 11. (a) The relation of Reduced Recall and due time; (b) the relation of Reduced Precision and due time
Next, since the classic Recall and Precision measures are related to the Due Time, a new metric is needed to evaluate
the performance of various waiting strategies. We use the Reduced Recall and the Reduced Precision measures to
present the results of Figure 10, as shown in Figure 11. We found that the two reduced measures take Due Time into
216