Download - Educational Technology & Society

More documents

Recommendations

Info

Another similar evaluation was reported in (Miller & Butz 2004). This study has evaluated the usability and effectiveness of Interactive Multimedia Intelligent System (IMITS), a system designed to tutor second year electrical engineering undergraduates. IMITS was evaluated from two perspectives, usability of the software and effectiveness of the software. The Usability Questionnaire was used to gather information concerning students’ reactions to the software. Also, usability data was obtained from the system’s log files. Similar technique as in DEPTHS was used to examine the impact of IMITS on student learning. The authors have used a quasi-experimental design, with a control and an experimental group of students. Analysis of variance (ANOVA) was used to test the hypothesis that the students who were using IMITS learned more than their counterparts under control conditions. Overall, students’ responses were favorable. IMITS improved performance on at least one classroom achievement measure. Regression analyses revealed that the more students used IMITS to learn some engineering concept (with usage defined as percentage of the questions encountered on a particular engineering concept), the better they learned that concept. Comparing to IMITS, we used the similar evaluation design in DEPTHS. We have reported on the systems usability and its effectiveness. Both evaluations have shown that the students’ responses were favorable. However, we go step further in evaluation of the system effectiveness. Contrary to IMITS evaluation of DEPTHS introduce a comparison of pre-test and post-test results in experimental group that lacks in IMITS. Our approach to evaluation of DEPTHS was influenced by the work reported in (Barker et al. 2002). They have applied a qualitative approach when evaluating their adaptive multimedia tutoring system. The authors have pointed out that an evaluation in real-world settings presents so many uncontrollable variables that traditional quantitative methods can give misleading results. They have suggested combining qualitative and quantitative analysis and have shown the advantages of this combined approach in real-world contexts. Their quantitative ANOVA results have shown that certain differences in mean values are unlikely to stem from random variations among students. Such strong rejection of the random-variations hypothesis would be impossible with a purely qualitative analysis. While agreeing with this approach, we have also adopted Kirkpatrick’s model of evaluation as a foundation of our evaluation design in DEPTHS. We found an interesting approach for evaluating the accuracy of the student model in (Millan & Perez de la Cruz 2002). The approach is based on the usage of simulated students since the cognitive state of a simulated student, unlike that of a real student, can be precisely determined, so the evaluation results are more accurate. However, as the authors pointed out, simulations could not provide the same degree of validity as experiments with real human students. This work is a good example of separating the evaluation of the accuracy of a student model from the evaluation of the adaptations efficiency based on the student model. By evaluating an intelligent tutoring system modularly with simulated students, the effects of each component can be separated from one another and the weakest component(s) could be easily identified and improved in order to increase the effectiveness of an adaptive system. In the future, we are planning to apply the similar approach with DEPTHS, in order to evaluate its modules, namely student model, pedagogical module, as well as the accuracy of student model assessment and adaptation effectiveness. We believe that this approach could give us valuable results on design and efficiency of each DEPTHS’s module. Conclusions This paper presents the evaluation of DEPTHS, an intelligent tutoring system (ITS) for teaching/learning software design patterns. The adopted evaluation approach is quite general, and can be equally well applied for evaluation of other ITSs. In particular, we used first two levels (reaction and learning) from the well-known Kirkpatrick’s model (Kirkpatrick 1979). The conducted evaluation studies targeted primarily the effectiveness of the DEPTHS system as well as the accuracy of its assumptions about the students’ knowledge level. The semester-long evaluation study has provided us with insights into strengths and weaknesses of the DEPTHS system. It has also made clear directions for future actions. We reported several advantages of the DEPTHS system over the traditional approach to teaching/learning design patterns. Students who learned with DEPTHS found that the system helped them to learn a lot about design patterns. They were especially pleased with the system’s ability to provide them with many useful information, feedback messages and advice for further work. Students’ responses indicated the need for regular communication with teachers and other students as the underpinning priorities for successful completion of online learning. 127
To test the learning effectiveness of DEPTHS, we used t-test and compared the pre-test and post-test results of an experimental and two control groups. The statistical analysis showed that the change from the pre-test to the post-test results was greater in the experimental than in the control groups. These findings indicate that students who learned with DEPTHS performed better than students who learned in the traditional way and that learning with DEPTHS brings in improvements in performance over time. We have also applied the One-way ANOVA statistical model to test for differences among the three study groups. The outcome of this test was that we accepted our null hypothesis that the students in the experimental group will perform as well as students who learn in the traditional way. This finding is obviously inconsistent with the result of the t-test, and another confirmation of the difficulty of accurately measuring the effectiveness of a certain educational tool on the students’ performance. However, we are encouraged with the fact that this system even in its early stages has better results than traditional learning. Finally, we compared the students’ knowledge of the domain as captured in their student models with their results on post-test in order to evaluate DEPTHS’s ability to accurately assess student knowledge of the subject domain. We found that the proposed student model does reflect the students’ knowledge of the subject matter, regardless the observed slight difference between the end-test results and the student model. Regarding our future work, there are several directions to be explored. Our first goal concerning further improvements of the DEPTHS system includes enabling the Student Model to store not only student’s knowledge, but also students’ cognitive and behavioral characteristics, such as memory capacity and motivation. This data will facilitate even more effective adaptation of the learning material. We are also planning to support interactions among students and the tutor in a group discussion environment, as well as to enable integration of additional supporting software tools, such as ArgoUML (http://argouml.tigris.org/). Regarding that, we are currently developing a Semantic web-based framework that integrates a set of existing tools in order to enhance students learning experience when collaboratively learning about Software patterns on the Web (Jeremic et al. 2008). References Antheil, J.H., & Casper, I.G. (1986). Comprehensive evaluation model: A tool for the evaluation of nontraditional educational programs. Innovative Higher Education, 11 (1), 55-64. Barker, T., Jones, S., Britton, C., & Messer, D. (2002). The Use of a Co-operative Student Model of Learner Characteristics to Configure a Multimedia Application. User Modeling and User-Adapted Interaction, 12 (2-3), 207- 241. Beaumont, I. (1994). User modeling in the interactive anatomy tutoring system ANATOMTUTOR. User Models and User Adapted Interaction 4 (1), 21-45. Brown, E., Stewart, C.D., Tim, J.B. (2006). Adapting for Visual and Verbal Learning Styles in AEH. Proceedings of the 6 th IEEE ICALT Conference, CA: IEEE Computer <strong>Society</strong>, 1145-1146. Brusilovsky, P. (1996). Methods and Techniques of Adaptive Hypermedia. User Modeling and User-Adapted Interaction, 6, 87-129. Burton, R.R., & Brown, J.S. (1976). A tutoring and student modeling paradigm for gaming environments. ACM SIGCSE Bulletin, 8 (1), 236-246. Davies, P.L. (2000). The one-way analysis of variance, retrieved March 8, 2009 from http://www.stat-math.uniessen.de/~davies/one-way.ps.gz. Devedzić, V. (2001). Knowledge Modeling – State of the Art. Integrated Computer-Aided Engineering, 8 (3), 257- 281. Dixon, N.M. (1990). The Relationship Between Trainee Responses on Participant Reaction Forms and Posttest Scores. Human Resource Development Quarterly, 1 (2), 129-137. Endres, G.J., & Kleiner, B.H. (1990). How to measure management training and development effectiveness. Journal of European Industrial Training, 14 (9), 3-7. Harrer, A., & Devedzić, V. (2002). Design and analysis pattern in ITS architecture. Proceedings of the ICCE 2002 Conference, CA: IEEE Computer <strong>Society</strong>, 523-527. 128
Page 1 and 2:
April 2009 Volume 12 Number 2
Page 3 and 4:
Abstracting and Indexing Educationa
Page 5 and 6:
Engaging students in multimedia-med
Page 7 and 8:
this study discusses computer-based
Page 9 and 10:
Sampling for preliminary study One
Page 11 and 12:
Part 2 Pearson correlation .349** 1
Page 13 and 14:
Figure 1. Box and whisker plot of t
Page 15 and 16:
characteristics and learning styles
Page 17 and 18:
However, critical reflection and in
Page 19 and 20:
novels, so they collected any relat
Page 21 and 22:
1. Theories of teaching: Comments m
Page 23 and 24:
grammatical errors and basic writin
Page 25 and 26:
Ho, B., & Richards, J. C. (1993). R
Page 27 and 28:
Hwang, K.-A., & Yang, C.-H. (2009).
Page 29 and 30:
e deduced from the Affective Domain
Page 31 and 32:
Detection procedure Figure 1 shows
Page 33 and 34:
avoided. Image processing is perfor
Page 35 and 36:
falling asleep) as defined in this
Page 37 and 38:
classmates or left their seats duri
Page 39 and 40:
help teachers to recognize the lear
Page 41 and 42:
Hsieh, P.-H., & Dwyer, F. (2009). T
Page 43 and 44:
significant comprehension effects o
Page 45 and 46:
Treatment 3 (keyword group): Studen
Page 47 and 48:
Criteria of achievement measures Th
Page 49 and 50:
esults. A correlational analysis de
Page 51 and 52:
A 2 x 1 ANOVA analyzed the effect o
Page 53 and 54:
References Aragon, S. R. (2004). In
Page 55 and 56:
Raphael, T. (1982). Improving quest
Page 57 and 58:
educing maintenance costs. The cont
Page 59 and 60:
Synchronization functions According
Page 61 and 62:
inconsistency) and navigate directl
Page 63 and 64:
the suitability of the instructiona
Page 65 and 66:
Design of manipulation process The
Page 67 and 68:
Table 1: Comparative analysis of au
Page 69 and 70:
the convenience of mobile-based ass
Page 71 and 72:
According to evaluation result, the
Page 73 and 74:
Tan, O. S., Parsons, R. D., Hinson,
Page 75 and 76:
At a deeper level, though, there wa
Page 77 and 78:
theoretical assumptions. Two theore
Page 79 and 80:
The students were not going to have
Page 81 and 82: slightly different. The group who c
Page 83 and 84: every item, not giving any result.
Page 85 and 86: edited and presented to the class u
Page 87 and 88: Figure 10. Screen captures from the
Page 89 and 90: Some of the concerns that emerge fr
Page 91 and 92: Ajayi, L. (2009). An Exploration of
Page 93 and 94: As a result of the confluence of di
Page 95 and 96: peers (Schellens et al. 2005). Sche
Page 97 and 98: The integration of discussion board
Page 99 and 100: ecause the technology allowed the p
Page 101 and 102: The pre-service teachers perceived
Page 103 and 104: Furthermore, the study demonstrated
Page 105 and 106: Schellens, T., van Keer, H., & Valc
Page 107 and 108: McKean, 2003). Instructors will see
Page 109 and 110: program at The Ohio State Universit
Page 111 and 112: Knowledge and skills of developing
Page 113 and 114: their monthly reflections or assign
Page 115 and 116: Bull, K., Montgomery, D., Overton,
Page 117 and 118: frustrating the student). ITSs make
Page 119 and 120: • Link removal - advanced student
Page 121 and 122: Expert Module Expert Module’s mai
Page 123 and 124: Based on the questionnaire results
Page 125 and 126: students were told that their perfo
Page 127 and 128: thirds of the interviewed students
Page 129 and 130: Throughout the semester, students i
Page 131: human teacher or question developer
Page 135 and 136: APPENDIX A Computer Science and Sof
Page 137 and 138: Thus, the use of computers changes
Page 139 and 140: Specifically, two objectives were p
Page 141 and 142: Nonetheless, both teachers assign a
Page 143 and 144: 48% 35% 53% similar similar 69% 42%
Page 145 and 146: Table 4: Students’ attitudes Stat
Page 147 and 148: activities. Specifically, the histo
Page 149 and 150: Huang, Y.-M., Huang, T.-C., Wang, K
Page 151 and 152: • Could recommended learning sequ
Page 153 and 154: Analysis of sequences prediction me
Page 155 and 156: Figure 4. The sorting of learning s
Page 157 and 158: The figures 5a and 5b illustrate th
Page 159 and 160: Learning sequence recommendation sy
Page 161 and 162: In this study, forty subjects were
Page 163 and 164: The potential for LSRS to promote l
Page 165 and 166: They pointed out that LSRS helped t
Page 167 and 168: Cover, T. M., & Thomas, J. A. (1991
Page 169 and 170: Related Efforts An Internet forum i
Page 171 and 172: with handheld devices and they were
Page 173 and 174: agent would direct learners to a le
Page 175 and 176: Mobile RSS Aggregator In order to s
Page 177 and 178: learning, such as posting question
Page 179 and 180: With novel technological support, m
Page 181 and 182: Shen, L., Wang, M., & Shen, R. (200
Page 183 and 184:
as an ‘affective loop’, which r
Page 185 and 186:
Figure 2 is an example of two-dimen
Page 187 and 188:
preference was gathered through dat
Page 189 and 190:
compute the heart rate (HR) as a fu
Page 191 and 192:
sessions. Of all the 18 learning se
Page 193 and 194:
Burleson, W., Picard, R. W., Perlin
Page 195 and 196:
Wu, C.-C., & Lai, C.Y. (2009). Wire
Page 197 and 198:
The wireless handheld learning envi
Page 199 and 200:
the lengthiness of the scales, whic
Page 201 and 202:
Nursing dictionary We constructed a
Page 203 and 204:
them. The computers were networked
Page 205 and 206:
“The major difference was that I
Page 207 and 208:
questionnaire. We do not consider t
Page 209 and 210:
Faruque, F., Hewlett, P. O., Wyatt,
Page 211 and 212:
etween a user and the system. Addit
Page 213 and 214:
Figure 2. A concept map representin
Page 215 and 216:
C5.0 can help teachers to generate
Page 217 and 218:
that a selected programming exercis
Page 219 and 220:
differences between our system and
Page 221 and 222:
1(students can complete without any
Page 223 and 224:
The PADS sometimes assigned very di
Page 225 and 226:
Corno, L. (2000). Looking at Homewo
Page 227 and 228:
Hwang, W.-Y., Hsu, J.-L., Tretiakov
Page 229 and 230:
listed these functions as allowing
Page 231 and 232:
In the research presented in this a
Page 233 and 234:
Research Tools Web-based learning e
Page 235 and 236:
Table 1. Operational definitions of
Page 237 and 238:
action, interaction, and outeractio
Page 239 and 240:
for them was getting the answer as
Page 241 and 242:
Table 6 shows the learners’ prefe
Page 243 and 244:
Bell, P., & Winn, W. (2000). Distri
Page 245 and 246:
Askar, P., & Altun, A. (2009). CogS
Page 247 and 248:
elations, interactions and activiti
Page 249 and 250:
Figure 3. A comparison of knowledge
Page 251 and 252:
ontology will easily be extensible
Page 253 and 254:
Figure 7. Visual Representation of
Page 255 and 256:
Step 5: Use Boolean to combine the
Page 257 and 258:
2) CogSkillNet provides classroom i
Page 259 and 260:
Neo, M., & Neo, T.-K. (2009). Engag
Page 261 and 262:
• Conversation and collaboration
Page 263 and 264:
IMAGE 2 2. Problem identification I
Page 265 and 266:
Methodology At the end of the proje
Page 267 and 268:
presentation skills (m = 3.72), and
Page 269 and 270:
6. Can’t be denying that, sometim
Page 271 and 272:
Lambert, N. M., & McCombs, B. J. (1
Page 273 and 274:
not detected in a previous study, t
Page 275 and 276:
et al. (2001) studied the learning
Page 277 and 278:
all the students. At the midpoint o
Page 279 and 280:
The observed pairs were aware of be
Page 281 and 282:
Table 2 shows the results from a mo
Page 283 and 284:
Table 5: The distribution of time (
Page 285 and 286:
As discussed in the section on prev
Page 287 and 288:
McDowell, C., Werner, L., Bullock,
Page 289 and 290:
Comprehension monitoring is the awa
Page 291 and 292:
or roles in a context, such as “I
Page 293 and 294:
Correlation coefficient was also co
Page 295 and 296:
(a) Text 1 (b) Text 2 (c) Text 3 (d
Page 297 and 298:
Table 3 shows that the average read
Page 299 and 300:
Figure 10. Trace results of the les
Page 301 and 302:
monitor their reading process and h
Page 303 and 304:
Conn, S. R., Roberts, R. L., & Powe
Page 305 and 306:
The technology-mediated groups, wit
Page 307 and 308:
hybrid model of supervision was pos
Page 309 and 310:
Our research indicates that the hyb
Page 311 and 312:
Janoff, D. S., & Schoenholtz-Read,
Page 313 and 314:
goal of using storyboards. But, thi
Page 315 and 316:
In general, learning activities nee
Page 317 and 318:
Furthermore, free subjects, such as
Page 319 and 320:
Storyboarding Roots and related wor
Page 321 and 322:
Referees, on the other hand, may wa
Page 323 and 324:
Symbols Interpretation Defines a un
Page 325 and 326:
Figure 4. Top-level storyboard for
Page 327 and 328:
may take subjects such as practical
Page 329 and 330:
network security information envir
Page 331 and 332:
The concrete procedure and interact
Page 333 and 334:
(e.g., for graduation) using such i
Page 335 and 336:
Thus, academic education modeling b
Page 337 and 338:
Feyer, T., Kao, O., Schewe, K.-D.,
Page 339 and 340:
Richards, G. (2009). Book review: T
Page 341 and 342:
expands the child’s proximal envi
Page 343:
Chapter 6 exemplifies the activitie
show all

Download - Educational Technology & Society

Create successful ePaper yourself

Delete template?

Save as template?