Download - Educational Technology & Society

Having applied the One-way ANOVA model on the pre-test results, we found that these scores did not differ much
between the three groups (f=0.108, df=2, P= 0.89). Table 2 presents the results of applying the One-way ANOVA
model on the post-test results. Post-test scores among three groups showed no significant differences (f=1.18, df=2,
P=0.32), as well (Brown et al 2006). As F-value is lower than the F-critical value, we concluded that statistically
there is no significant difference, and the score differences could be explained the best as being present by chance.
The outcome of our test was that we accepted the null hypothesis that the students in the experimental group will
perform in the similar manner as students who learned in the traditional way (groups B and C).
Even though the results of One-way ANOVA can bring conclusion that DEPTHS does not lead to significant
improvements over the traditional learning, we are encouraged with the fact that our system even in its early stages
has better results than traditional learning. In particular, the results of t-test performed on the experimental group
have shown significant progress of those students who learned with DEPTHS. Moreover, the qualitative results
however indicate that the students prefer this kind of learning over the traditional learning of design patterns. Our
experiences from this evaluation will lead to many improvements of the system which will make it far more
successful in the future.
Student model assessment evaluation
To evaluate the accuracy of the DEPTHS’s student knowledge assessment we compared the students’ knowledge of
the domain as captured in their student models with their results on an external test, performed after the course was
completed. For each student, the DEPTHS system keeps track of all the tests he has solved throughout the course
(i.e., after having learned each domain concept) and evaluates his overall knowledge of the domain from the tests’
results. The external test was designed by the group of teachers, experts in the domain of software engineering. It
was given to the students at the end of the course to assess their knowledge of the whole course content.
The results of the 14 participants from the experimental group were analyzed. The questions of the external test were
divided into 10 groups, each group covering one domain concept, so that, based on the test results, we could assess
the students’ knowledge of each concept individually. Comparing the results of the external test with the student
model data (Fig. 8), we found that the external test shows slightly lower knowledge level (15.43%). That difference
was expected, because the system (i.e. its student modeling component) assesses student knowledge on each concept
immediately after the student has learned it, whereas the external test assesses student knowledge at the end of the
whole course when a number of other variables affect the final results. For example, students could not use DEPTHS
out of the classroom, but we could not control if they are using some other sources of information (literature or the
Internet). Moreover, when estimating the student’s knowledge of a concept, the system takes the best score over all
the test attempts related to that concept. However, if instead of taking the best result we toke the average test result
for a concept (from system’s logs on each concept), we get results that are 11.37% lower than those on the external
test. As the result of the student model assessment evaluation performed on DEPTHS, we finally concluded that the
comparison between the external test and student model could not be realistic indicator of the student model
accuracy, as it is performed during the long period of time (one semester), while system assesses student knowledge
immediately after the learning of each concept. Therefore, we decided to analyze the system’s log data in order to get
some more information about the student model used.
By analyzing the system’s log data we found that at the beginning of the course students tend to spent more time on
each lesson, give more attempts at quizzing and were often revisiting previously learned concepts. The majority of
the students (87%) attempted to solve the tests more than once during the first quarter of the course, and 34% gave
three or more attempts. This trend was diminishing as the course proceeded. At the last quarter, 49% of the students
made two or more attempts and only 11% tried to do the test more than twice. We think that this decrease in the
number of test reatempts is due to the fact that as the end of the semester was approaching, the students were having
increasingly more obligations on other courses and less time to dedicate to learning with DEPTHS. This is a well
known phenomenon, present at the Military Academy for years.
We found an additional interesting issue by analyzing the system’s log data. Some questions, annotated by their
creators as very easy, were not performed well by students. Contrary, some other questions described as very
difficult were surprisingly well done. Having analyzed this issue, we concluded that it originates from the fact that
the Diagnosis module uses data about a question’s difficulty level and time necessary to solve it provided by a
125