Download - Educational Technology & Society

More documents

Recommendations

Info

Laakso, M.-J., Myller, N., & Korhonen, A. (2009). Comparing Learning Performance of Students Using Algorithm Visualizations Collaboratively on Different Engagement Levels. Educational Technology & Society, 12 (2), 267–282. Comparing Learning Performance of Students Using Algorithm Visualizations Collaboratively on Different Engagement Levels Mikko-Jussi Laakso 1 , Niko Myller 2 and Ari Korhonen 3 1 Department of Information Technology, University of Turku, 22014 Turun Yliopisto, Turku, Finland // milaak@utu.fi // Tel +358 2 333 8672 // Fax +358 2 333 8600 2 Department of Computer Science and Statistics, University of Joensuu, P.O. Box 111, FI-80101 Joensuu, Joensuu, Finland // nmyller@cs.joensuu.fi // Tel +358 13 251 7929 // Fax +358 13 251 7955 3 Department of Computer Science and Engineering, Helsinki University of Technology, P.O. Box 5400, FI-02015 TKK, Espoo, Finland // archie@cs.hut.fi // Tel +358 9 451 3387 // Fax +358 9 451 3293 ABSTRACT In this paper, two emerging learning and teaching methods have been studied: collaboration in concert with algorithm visualization. When visualizations have been employed in collaborative learning, collaboration introduces new challenges for the visualization tools. In addition, new theories are needed to guide the development and research of the visualization tools for collaborative learning. We present an empirical study, in which learning materials containing visualizations on different Extended Engagement Taxonomy levels were compared, when students were collaboratively learning concepts related to binary heap. In addition, the students’ activities during the controlled experimental study were also recorded utilizing a screen capturing software. Pre- and post-tests were used as the test instruments in the experiment. No statistically significant differences were found in the post-test between the randomized groups. However, screen capturing and voice recording revealed that despite the randomization and instructions given to the students, not all of the students performed on the engagement level, to which they were assigned. By regrouping the students based on the monitored behavior, statistically significant differences were found in the total and pair average of the post-test scores. This confirms some of the hypothesis presented in the (Extended) Engagement Taxonomy. Keywords Algorithm visualization, Algorithm simulation, collaborative learning, Engagement taxonomy Introduction Since its introduction, it has been hoped that Algorithm Visualization (AV) would solve problems related to learning of data structures and algorithms. However, empirical evaluations have yielded mixed results when determining the usefulness of such visualizations as teaching and learning aids over traditional methods (see the meta-analysis of the research on AV by Hundhausen et al. (2002)). Thus, researchers have sought explanations for the mixed results as well as better grounds to justify the use of visualizations in teaching. Hundhausen et al. (2002) concluded that the activities performed by the students are more important than the content of the visualization. This has led to the analysis of different engagement levels Naps et al. (2002) by ITiCSE Working Group that proposed Engagement Taxonomy (ET) to describe the various types of activities that students perform with visualizations and their effect on learning and Myller et al. (in press) have developed it further into Extended Engagement Taxonomy (EET). Collaboration has become accepted and popular in Computer Science education. A good example is the benefits of pair programming (Nagappan et al., 2003; Williams et al., 2000; McDowell et al., 2003). Whilst visualizations are employed in collaborative learning, collaboration introduces new challenges for the visualization tools. For example, the exchange of experiences and ideas, and coordination of the joint work are needed when students are not working individually anymore (Suthers and Hundhausen, 2003). Furthermore, visualizations can provide a shared external memory that can initiate negotiations of meanings and act as a reference point when ideas are explained or misunderstandings are resolved (Suthers and Hundhausen, 2003). This implies that also new theories are needed to guide the development and research of the visualization tools for collaborative learning. In this paper, the applicability of EET in collaborative use of visualizations has been studied. We test the impact of EET levels on the performance when visualizations are used in collaboration. We present an empirical study, in which learning materials containing visualizations on different EET levels were compared when student pairs were collaboratively learning concepts related to binary heap. The pairs had a mutual task to read through a tutorial including visualizations and answer questions related to the topic. Although, statistically significant differences were ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 267
not detected in a previous study, the results indicated that the engagement level of the visualizations has an effect on the performance when students are working in pairs (Myller et al., 2007). Thus, we replicated that study in a different institution, and improved the settings in such a way that the detection of the statistically significant differences would be possible. In this paper, we report the results from the replication study conducted at the Helsinki University of Technology in which two groups of students were randomized to the computer lab sessions. Each session was randomly assigned to an EET level, either changing or controlled viewing (in the rest of the paper this can be also shortened to viewing when we are discussing about the groups), with the limitation that parallel sessions belonged to different conditions. During the analysis of the screen and voice recordings collected in the study, it was detected that despite the randomization and instructions given to the students, not all of the students performed their learning on the expected EET level. This meant that although the tool allowed students to learn on a higher EET level, some of the students choose not to do so, but worked on a lower engagement level. Fortunately, the screen capturing and voice recording done during the students’ learning process provided us a tool for noticing this and taking it into account in the analysis. Thus, in addition to the results from the study, we learned an important methodological lesson as well. Screen capturing and voice recording should be a standard procedure, because otherwise we cannot know for sure if the participants really do what we expect them to do. In Chapter 2, we describe the relevant literature related to the engagement taxonomy and similar theories. In addition, we give an overview of the learning tool used in the experiments. Chapter 3 describes the research setting, i.e., the used pre- and post-tests, subjects, materials, and procedures. In Chapter 4, we report on the results. Finally, in Chapters 5 and 6, we make conclusions and highlight some future directions. Previous Research Visualizations and Engagement As an attempt to describe the mixed results of previous research in AV usage (cf. (Hundhausen et al., 2002)) in learning and teaching of algorithms and data structures, Engagement Taxonomy (ET) was introduced by Naps et al. (2002). The central idea of the taxonomy is that the higher the engagement between the learner and the visualization, the higher the positive effects on learning outcomes. ET consists of six levels of engagement between the user and the visualization: No viewing There is no visualization to be viewed. Viewing The visualization is only looked at without any interaction. Responding Visualization is accompanied with questions, which are related to the content of the visualization. Changing Modification of the visualization is allowed, for example, by varying the input data set or algorithm simulation. Constructing Visualization of program or algorithm is created. Presenting Visualizations are presented to others for feedback and discussions. ET has been used in the development of AV tools and several studies have utilized the framework and provided further support for it (see, e.g., Grissom et al. (2003); Naps and Grissom (2002)). However, the time to study the materials on different ET levels has commonly been an uncontrolled variable in the studies, meaning that students have had freedom to use as little or as much time as they wanted to. Thus, those students who have been studying with visualizations that are on the higher ET level have spent more time on the task. This, in turn, makes it questionable if the reason for better performance in the post-test is due to the additional time spent on studying or the higher ET level of the materials. In the experiment, which is presented in this paper, we controlled the time so that all the students needed to spend exactly the same amount of time on learning the topic. There are also other studies which have shown that visualizations improve learning, without actually utilizing the ET framework in the design of the study (Ben-Bassat Levy et al., 2003). In addition to this, research in educational psychology and multimedia learning had also had similar results (Evans and Gibbons, 2006). 268
Page 1 and 2:
April 2009 Volume 12 Number 2
Page 3 and 4:
Abstracting and Indexing Educationa
Page 5 and 6:
Engaging students in multimedia-med
Page 7 and 8:
this study discusses computer-based
Page 9 and 10:
Sampling for preliminary study One
Page 11 and 12:
Part 2 Pearson correlation .349** 1
Page 13 and 14:
Figure 1. Box and whisker plot of t
Page 15 and 16:
characteristics and learning styles
Page 17 and 18:
However, critical reflection and in
Page 19 and 20:
novels, so they collected any relat
Page 21 and 22:
1. Theories of teaching: Comments m
Page 23 and 24:
grammatical errors and basic writin
Page 25 and 26:
Ho, B., & Richards, J. C. (1993). R
Page 27 and 28:
Hwang, K.-A., & Yang, C.-H. (2009).
Page 29 and 30:
e deduced from the Affective Domain
Page 31 and 32:
Detection procedure Figure 1 shows
Page 33 and 34:
avoided. Image processing is perfor
Page 35 and 36:
falling asleep) as defined in this
Page 37 and 38:
classmates or left their seats duri
Page 39 and 40:
help teachers to recognize the lear
Page 41 and 42:
Hsieh, P.-H., & Dwyer, F. (2009). T
Page 43 and 44:
significant comprehension effects o
Page 45 and 46:
Treatment 3 (keyword group): Studen
Page 47 and 48:
Criteria of achievement measures Th
Page 49 and 50:
esults. A correlational analysis de
Page 51 and 52:
A 2 x 1 ANOVA analyzed the effect o
Page 53 and 54:
References Aragon, S. R. (2004). In
Page 55 and 56:
Raphael, T. (1982). Improving quest
Page 57 and 58:
educing maintenance costs. The cont
Page 59 and 60:
Synchronization functions According
Page 61 and 62:
inconsistency) and navigate directl
Page 63 and 64:
the suitability of the instructiona
Page 65 and 66:
Design of manipulation process The
Page 67 and 68:
Table 1: Comparative analysis of au
Page 69 and 70:
the convenience of mobile-based ass
Page 71 and 72:
According to evaluation result, the
Page 73 and 74:
Tan, O. S., Parsons, R. D., Hinson,
Page 75 and 76:
At a deeper level, though, there wa
Page 77 and 78:
theoretical assumptions. Two theore
Page 79 and 80:
The students were not going to have
Page 81 and 82:
slightly different. The group who c
Page 83 and 84:
every item, not giving any result.
Page 85 and 86:
edited and presented to the class u
Page 87 and 88:
Figure 10. Screen captures from the
Page 89 and 90:
Some of the concerns that emerge fr
Page 91 and 92:
Ajayi, L. (2009). An Exploration of
Page 93 and 94:
As a result of the confluence of di
Page 95 and 96:
peers (Schellens et al. 2005). Sche
Page 97 and 98:
The integration of discussion board
Page 99 and 100:
ecause the technology allowed the p
Page 101 and 102:
The pre-service teachers perceived
Page 103 and 104:
Furthermore, the study demonstrated
Page 105 and 106:
Schellens, T., van Keer, H., & Valc
Page 107 and 108:
McKean, 2003). Instructors will see
Page 109 and 110:
program at The Ohio State Universit
Page 111 and 112:
Knowledge and skills of developing
Page 113 and 114:
their monthly reflections or assign
Page 115 and 116:
Bull, K., Montgomery, D., Overton,
Page 117 and 118:
frustrating the student). ITSs make
Page 119 and 120:
• Link removal - advanced student
Page 121 and 122:
Expert Module Expert Module’s mai
Page 123 and 124:
Based on the questionnaire results
Page 125 and 126:
students were told that their perfo
Page 127 and 128:
thirds of the interviewed students
Page 129 and 130:
Throughout the semester, students i
Page 131 and 132:
human teacher or question developer
Page 133 and 134:
To test the learning effectiveness
Page 135 and 136:
APPENDIX A Computer Science and Sof
Page 137 and 138:
Thus, the use of computers changes
Page 139 and 140:
Specifically, two objectives were p
Page 141 and 142:
Nonetheless, both teachers assign a
Page 143 and 144:
48% 35% 53% similar similar 69% 42%
Page 145 and 146:
Table 4: Students’ attitudes Stat
Page 147 and 148:
activities. Specifically, the histo
Page 149 and 150:
Huang, Y.-M., Huang, T.-C., Wang, K
Page 151 and 152:
• Could recommended learning sequ
Page 153 and 154:
Analysis of sequences prediction me
Page 155 and 156:
Figure 4. The sorting of learning s
Page 157 and 158:
The figures 5a and 5b illustrate th
Page 159 and 160:
Learning sequence recommendation sy
Page 161 and 162:
In this study, forty subjects were
Page 163 and 164:
The potential for LSRS to promote l
Page 165 and 166:
They pointed out that LSRS helped t
Page 167 and 168:
Cover, T. M., & Thomas, J. A. (1991
Page 169 and 170:
Related Efforts An Internet forum i
Page 171 and 172:
with handheld devices and they were
Page 173 and 174:
agent would direct learners to a le
Page 175 and 176:
Mobile RSS Aggregator In order to s
Page 177 and 178:
learning, such as posting question
Page 179 and 180:
With novel technological support, m
Page 181 and 182:
Shen, L., Wang, M., & Shen, R. (200
Page 183 and 184:
as an ‘affective loop’, which r
Page 185 and 186:
Figure 2 is an example of two-dimen
Page 187 and 188:
preference was gathered through dat
Page 189 and 190:
compute the heart rate (HR) as a fu
Page 191 and 192:
sessions. Of all the 18 learning se
Page 193 and 194:
Burleson, W., Picard, R. W., Perlin
Page 195 and 196:
Wu, C.-C., & Lai, C.Y. (2009). Wire
Page 197 and 198:
The wireless handheld learning envi
Page 199 and 200:
the lengthiness of the scales, whic
Page 201 and 202:
Nursing dictionary We constructed a
Page 203 and 204:
them. The computers were networked
Page 205 and 206:
“The major difference was that I
Page 207 and 208:
questionnaire. We do not consider t
Page 209 and 210:
Faruque, F., Hewlett, P. O., Wyatt,
Page 211 and 212:
etween a user and the system. Addit
Page 213 and 214:
Figure 2. A concept map representin
Page 215 and 216:
C5.0 can help teachers to generate
Page 217 and 218:
that a selected programming exercis
Page 219 and 220:
differences between our system and
Page 221 and 222: 1(students can complete without any
Page 223 and 224: The PADS sometimes assigned very di
Page 225 and 226: Corno, L. (2000). Looking at Homewo
Page 227 and 228: Hwang, W.-Y., Hsu, J.-L., Tretiakov
Page 229 and 230: listed these functions as allowing
Page 231 and 232: In the research presented in this a
Page 233 and 234: Research Tools Web-based learning e
Page 235 and 236: Table 1. Operational definitions of
Page 237 and 238: action, interaction, and outeractio
Page 239 and 240: for them was getting the answer as
Page 241 and 242: Table 6 shows the learners’ prefe
Page 243 and 244: Bell, P., & Winn, W. (2000). Distri
Page 245 and 246: Askar, P., & Altun, A. (2009). CogS
Page 247 and 248: elations, interactions and activiti
Page 249 and 250: Figure 3. A comparison of knowledge
Page 251 and 252: ontology will easily be extensible
Page 253 and 254: Figure 7. Visual Representation of
Page 255 and 256: Step 5: Use Boolean to combine the
Page 257 and 258: 2) CogSkillNet provides classroom i
Page 259 and 260: Neo, M., & Neo, T.-K. (2009). Engag
Page 261 and 262: • Conversation and collaboration
Page 263 and 264: IMAGE 2 2. Problem identification I
Page 265 and 266: Methodology At the end of the proje
Page 267 and 268: presentation skills (m = 3.72), and
Page 269 and 270: 6. Can’t be denying that, sometim
Page 271: Lambert, N. M., & McCombs, B. J. (1
Page 275 and 276: et al. (2001) studied the learning
Page 277 and 278: all the students. At the midpoint o
Page 279 and 280: The observed pairs were aware of be
Page 281 and 282: Table 2 shows the results from a mo
Page 283 and 284: Table 5: The distribution of time (
Page 285 and 286: As discussed in the section on prev
Page 287 and 288: McDowell, C., Werner, L., Bullock,
Page 289 and 290: Comprehension monitoring is the awa
Page 291 and 292: or roles in a context, such as “I
Page 293 and 294: Correlation coefficient was also co
Page 295 and 296: (a) Text 1 (b) Text 2 (c) Text 3 (d
Page 297 and 298: Table 3 shows that the average read
Page 299 and 300: Figure 10. Trace results of the les
Page 301 and 302: monitor their reading process and h
Page 303 and 304: Conn, S. R., Roberts, R. L., & Powe
Page 305 and 306: The technology-mediated groups, wit
Page 307 and 308: hybrid model of supervision was pos
Page 309 and 310: Our research indicates that the hyb
Page 311 and 312: Janoff, D. S., & Schoenholtz-Read,
Page 313 and 314: goal of using storyboards. But, thi
Page 315 and 316: In general, learning activities nee
Page 317 and 318: Furthermore, free subjects, such as
Page 319 and 320: Storyboarding Roots and related wor
Page 321 and 322: Referees, on the other hand, may wa
Page 323 and 324:
Symbols Interpretation Defines a un
Page 325 and 326:
Figure 4. Top-level storyboard for
Page 327 and 328:
may take subjects such as practical
Page 329 and 330:
network security information envir
Page 331 and 332:
The concrete procedure and interact
Page 333 and 334:
(e.g., for graduation) using such i
Page 335 and 336:
Thus, academic education modeling b
Page 337 and 338:
Feyer, T., Kao, O., Schewe, K.-D.,
Page 339 and 340:
Richards, G. (2009). Book review: T
Page 341 and 342:
expands the child’s proximal envi
Page 343:
Chapter 6 exemplifies the activitie
show all

Download - Educational Technology & Society

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?