October 2007 Volume 10 Number 4 - Educational Technology ...

October 2007 

Volume 10 Number 4

Educational Technology & Society 

An International Journal 

Aims and Scope 

Educational Technology & Society is a quarterly journal published in January, April, July and October. Educational Technology & Society 

seeks academic articles on the issues affecting the developers of educational systems and educators who implement and manage such systems. The 

articles should discuss the perspectives of both communities and their relation to each other: 

• Educators aim to use technology to enhance individual learning as well as to achieve widespread education and expect the technology to blend 

with their individual approach to instruction. However, most educators are not fully aware of the benefits that may be obtained by proactively 

harnessing the available technologies and how they might be able to influence further developments through systematic feedback and 

suggestions. 

• Educational system developers and artificial intelligence (AI) researchers are sometimes unaware of the needs and requirements of typical 

teachers, with a possible exception of those in the computer science domain. In transferring the notion of a 'user' from the human-computer 

interaction studies and assigning it to the 'student', the educator's role as the 'implementer/ manager/ user' of the technology has been forgotten. 

The aim of the journal is to help them better understand each other's role in the overall process of education and how they may support each 

other. The articles should be original, unpublished, and not in consideration for publication elsewhere at the time of submission to Educational 

Technology & Society and three months thereafter. 

The scope of the journal is broad. Following list of topics is considered to be within the scope of the journal: 

Architectures for Educational Technology Systems, Computer-Mediated Communication, Cooperative/ Collaborative Learning and 

Environments, Cultural Issues in Educational System development, Didactic/ Pedagogical Issues and Teaching/Learning Strategies, Distance 

Education/Learning, Distance Learning Systems, Distributed Learning Environments, Educational Multimedia, Evaluation, Human-Computer 

Interface (HCI) Issues, Hypermedia Systems/ Applications, Intelligent Learning/ Tutoring Environments, Interactive Learning Environments, 

Learning by Doing, Methodologies for Development of Educational Technology Systems, Multimedia Systems/ Applications, Network-Based 

Learning Environments, Online Education, Simulations for Learning, Web Based Instruction/ Training 

Editors 

Kinshuk, Athabasca University, Canada; Demetrios G Sampson, University of Piraeus & ITI-CERTH, Greece; Ashok Patel, CAL Research 

& Software Engineering Centre, UK; Reinhard Oppermann, Fraunhofer Institut Angewandte Informationstechnik, Germany. 

Editorial Assistant 

Barbara Adamski, Athabasca University. 

Associate editors 

Nian-Shing Chen, National Sun Yat-sen University, Taiwan; Alexandra I. Cristea, Technical University Eindhoven, The Netherlands; John 

Eklund, Access Australia Co-operative Multimedia Centre, Australia; Vladimir A Fomichov, K. E. Tsiolkovsky Russian State Tech Univ, 

Russia; Olga S Fomichova, Studio "Culture, Ecology, and Foreign Languages", Russia; Piet Kommers, University of Twente, The 

Netherlands; Chul-Hwan Lee, Inchon National University of Education, Korea; Brent Muirhead, University of Phoenix Online, USA; 

Erkki Sutinen, University of Joensuu, Finland; Vladimir Uskov, Bradley University, USA. 

Advisory board 

Ignacio Aedo, Universidad Carlos III de Madrid, Spain; Luis Anido-Rifon, University of Vigo, Spain; Alfred Bork, University of 

California, Irvine, USA; Rosa Maria Bottino, Consiglio Nazionale delle Ricerche, Italy; Mark Bullen, University of British Columbia, 

Canada; Tak-Wai Chan, National Central University, Taiwan; Darina Dicheva, Winston-Salem State University, USA; Brian Garner, 

Deakin University, Australia; Roger Hartley, Leeds University, UK; Harald Haugen, Høgskolen Stord/Haugesund, Norway; J R Isaac, 

National Institute of Information Technology, India; Mohamed Jemni, University of Tunis, Tunisia; Paul Kirschner, Open University of the 

Netherlands, The Netherlands; William Klemm, Texas A&M University, USA; Rob Koper, Open University of the Netherlands, The 

Netherlands; Ruddy Lelouche, Universite Laval, Canada; David McConnell, Lancaster University, UK; Rory McGreal, Athabasca 

University, Canada; David Merrill, Brigham Young University - Hawaii, USA; Marcelo Milrad, Växjö University, Sweden; Riichiro 

Mizoguchi, Osaka University, Japan; Hiroaki Ogata, Tokushima University, Japan; Toshio Okamoto, The University of Electro- 

Communications, Japan; Thomas C. Reeves, The University of Georgia, USA; Gilly Salmon, University of Leicester, United Kingdom; 

Norbert M. Seel, Albert-Ludwigs-University of Freiburg, Germany; Timothy K. Shih, Tamkang University, Taiwan; Yoshiaki Shindo, 

Nippon Institute of Technology, Japan; Brian K. Smith, Pennsylvania State University, USA; J. Michael Spector, Florida State University, 

USA; Chin-Chung Tsai, National Taiwan University of Science and Technology, Taiwan; Stephen J.H. Yang, National Central University, 

Taiwan. 

Assistant Editors 

Sheng-Wen Hsieh, Far East University, Taiwan; Taiyu Lin, Massey University, New Zealand; Kathleen Luchini, University of Michigan, 

USA; Dorota Mularczyk, Independent Researcher & Web Designer; Carmen Padrón Nápoles, Universidad Carlos III de Madrid, Spain; 

Ali Fawaz Shareef, Massey University, New Zealand; Jarkko Suhonen, University of Joensuu, Finland. 

Executive peer-reviewers 

http://www.ifets.info/ 

Subscription Prices and Ordering Information 

For subscription information, please contact the editors at kinshuk@ieee.org. 

Advertisements 

Educational Technology & Society accepts advertisement of products and services of direct interest and usefulness to the readers of the journal, 

those involved in education and educational technology. Contact the editors at kinshuk@ieee.org. 

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the 

copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies 

are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by 

others ISSN than 1436-4522. IFETS must © International be honoured. Forum Abstracting of Educational with credit Technology is permitted. & To Society copy otherwise, (IFETS). The to republish, authors and to post the on forum servers, jointly or to retain redistribute the copyright to lists, of requires the articles. prior 

specific Permission permission to make and/or digital a fee. or hard Request copies permissions of part or all from of this the editors work for at personal kinshuk@ieee.org. or classroom use is granted without fee provided that copies are not made or distributed 

for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be 

honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a 

fee. Request permissions from the editors at kinshuk@massey.ac.nz. 

i

Abstracting and Indexing 

Educational Technology & Society is abstracted/indexed in Social Science Citation Index, Current Contents/Social & Behavioral Sciences, ISI 

Alerting Services, Social Scisearch, ACM Guide to Computing Literature, Australian DEST Register of Refereed Journals, Computing Reviews, 

DBLP, Educational Administration Abstracts, Educational Research Abstracts, Educational Technology Abstracts, Elsevier Bibliographic 

Databases, ERIC, Inspec, Technical Education & Training Abstracts, and VOCED. 

Guidelines for authors 

Submissions are invited in the following categories: 

• Peer reviewed publications: Full length articles (4000 - 7000 words) 

• Book reviews 

• Software reviews 

• Website reviews 

All peer review publications will be refereed in double-blind review process by at least two international reviewers with expertise in the relevant 

subject area. Book, Software and Website Reviews will not be reviewed, but the editors reserve the right to refuse or edit review. 

For detailed information on how to format your submissions, please see: 

http://www.ifets.info/guide.php 

Submission procedure 

Authors, submitting articles for a particular special issue, should send their submissions directly to the appropriate Guest Editor. Guest Editors 

will advise the authors regarding submission procedure for the final version. 

All submissions should be in electronic form. The editors will acknowledge the receipt of submission as soon as possible. 

The preferred formats for submission are Word document and RTF, but editors will try their best for other formats too. For figures, GIF and 

JPEG (JPG) are the preferred formats. Authors must supply separate figures in one of these formats besides embedding in text. 

Please provide following details with each submission: Author(s) full name(s) including title(s), Name of corresponding author, Job 

title(s), Organisation(s), Full contact details of ALL authors including email address, postal address, telephone and fax numbers. 

The submissions should be uploaded at http://www.ifets.info/ets_journal/upload.php. In case of difficulties, they can also be sent via email to 

(Subject: Submission for Educational Technology & Society journal): kinshuk@ieee.org. In the email, please state clearly that the manuscript is 

original material that has not been published, and is not being considered for publication elsewhere. 









ii

Journal of Educational Technology & Society 

Volume 10 Number 4 2007 

Table of contents 

Special issue articles 

Theme: Current Approaches to Network-Based Learning in Scandinavia 

Guest Editors: Marcelo Milrad and Per Flensburg 

Current Approaches to Network-Based Learning in Scandinavia (Guest Editorial) 

Marcelo Milrad and Per Flensburg 

Design and Use of Collaborative Network Learning Scenarios: The DoCTA Experience 

Barbara Wasson 

Dynamic Assessment and the “Interactive Examination” 

Anders Jönsson, Nikos Mattheos, Gunilla Svingby and Rolf Attström 

Participation in an Educational Online Learning Community 

Anders D. Olofsson 

Framing Work-Integrated e-Learning with Techno-Pedagogical Genres 

Lars Svensson and Christian Östlund 

Netlearning and Learning through Networks 

Mikael Wiberg 

Anytime, Anywhere Learning Supported by Smart Phones: Experiences and Results from the MUSIS 

Project 

Marcelo Milrad and Daniel Spikol 

Structuring and Regulating Collaborative Learning in Higher Education with Wireless Networks and 

Mobile Tools 

Sanna Järvelä, Piia Näykki, Jari Laru and Tiina Luokkanen 

Full length articles 

Implementation of an Improved Adaptive Testing Theory 

Mansoor Al-A'ali 

Measures of Partial Knowledge and Unexpected Responses in Multiple-Choice Tests 

Shao-Hua Chang, Pei-Chun Lin and Zih-Chuan Lin 

Standardization from Below: Science and Technology Standards and Educational Software 

Kenneth R. Fleischmann 

Factors Influencing Junior High School Teachers’ Computer-Based Instructional Practices Regarding 

Their Instructional Evolution Stages 

Ying-Shao Hsu, Hsin-Kai Wu and Fu-Kwun Hwang 

Analysis of Computer Teachers’ Online Discussion Forum Messages about their Occupational Problems 

Deniz Deryakulu and Sinan Olkun 

The learning computer: low bandwidth tool that bridges digital divide 

Russell Johnson, Elizabeth Kemp, Ray Kemp and Peter Blakey 

Reliability and Validity of Authentic Assessment in a Web Based Course 

Raimundo Olfos and Hildaura Zulantay 

Transforming classroom teaching & learning through technology: Analysis of a case study 

Rosa Maria Bottino and Elisabetta Robotti 



are ISSN not 1436-4522. made or distributed © International for profit Forum or commercial of Educational advantage Technology and that copies & Society bear the (IFETS). full citation The authors on the first and page. the forum Copyrights jointly for retain components the copyright of this work of the owned articles. by 

others Permission than to IFETS make must digital be or honoured. hard copies Abstracting of part or with all of credit this work is permitted. for personal To copy or classroom otherwise, use to is republish, granted without to post fee on provided servers, or that to copies redistribute are not to made lists, requires or distributed prior 

specific for profit permission or commercial and/or advantage a fee. Request and that permissions copies bear from the the full editors citation at on kinshuk@ieee.org. 

the first page. Copyrights for components of this work owned by others than IFETS must be 



1-2 

3-16 

17-27 

28-38 

39-48 

49-61 

62-70 

71-79 

80-94 

95-109 

110-117 

118-130 

131-142 

143-155 

156-173 

174-186 

iii

The Relationship of Kolb Learning Styles, Online Learning Behaviors and Learning Outcomes 

Hong Lu, Lei Jia, Shu-hong Gong and Bruce Clark 

In an Economy for Reusable Learning Objects, Who Pulls the Strings? 

Tim Linsey and Christopher Tompsett 

I Design; Therefore I Research: Revealing DBR through Personal Narrative 

Dave S. Knowlton 

Artificial Intelligence Approach to Evaluate Students’ Answerscripts Based on the Similarity Measure 

between Vague Sets 

Hui-Yu Wang and Shyi-Ming Chen 

A Web-based Educational Setting Supporting Individualized Learning, Collaborative Learning and 

Assessment 

Agoritsa Gogoulou, Evangelia Gouli, Maria Grigoriadou, Maria Samarakou and Dionisia Chinou 

Seven Problems of Online Group Learning (and Their Solutions) 

Tim S. Roberts and Joanne M. McInnerney 

Using Computers to Individually-generate vs. Collaboratively-generate Concept Maps 

So Young Kwon and Lauren Cifuentes 

Instructional Design for Best Practice in the Synchronous Cyber Classroom 

Megan Hastie, Nian-Shing Chen and Yen-Hung Kuo 

Book review(s) 

Cases on global e-learning practices: successes and pitfalls 

Reviewer: Richard Malinski 

Website review(s) 

WordChamp: Learn Language Faster 

Reviewer: Ferit Kılıçkaya 









187-196 

197-208 

209-223 

224-241 

242-256 

257-268 

269-280 

281-294 

295-297 

298-299 

iv

Milrad, M., & Flensburg, P. (2007). Current Approaches to Network-Based Learning in Scandinavia (Guest Editorial). 

Educational Technology & Society, 10 (4), 1-2. 

Current Approaches to Network-Based Learning in Scandinavia 

(Guest Editorial) 

Marcelo Milrad 

Center for Learning and Knowledge Technologies (CeLeKT), Växjö University, Sweden // 

marcelo.milrad@msi.vxu.se 

Per Flensburg 

Department of Economy and Informatics, University West, Sweden // per.flensburg@hv.se 

This special issue of Educational Technology & Society aims at giving the reader an overview of current 

Scandinavian research in network-based learning. The base of the articles described in this issue features a selection 

of the best research papers from the Netlearning 2006 conference held at the Blekinge Technical University, Sweden 

in May 2006. 

There is a long tradition regarding distance education in Scandinavia. The first form of this type of education was 

called folk high school (institutions of informal education for adults) and started in 1844 in Rødding, Denmark. The 

folk high school was created for adult people and most often it was carried out in the form of a boarding school. The 

main ideas behind the folk high school approach were formulated by N.F.S. Grundtvig (1783–1872); a Danish 

teacher, philosopher and pastor. Grundtvig's pedagogical ideas were focused on learners´ active participation and 

experimentation during their studies. More than fifty years later, the first correspondence study institute (Hermods) 

was created in Malmö, Sweden in 1898. The main idea of Hermods was to provide educational materials and 

feedback to students by mail. The Hermods institute still exists, although the traditional letter-based courses have 

been replaced by Internet-based materials. The concepts of learners´ involvement and active participation inspired by 

Grundtvig´s ideas are still central components in many of the current network-based learning approaches used in 

Scandinavia. Thus, each of the papers selected for this special issue could be seen as an extension and evolution of 

these ideas. 

The first paper in this collection, entitled “Design and Use of Collaborative Network Learning Scenarios: The 

DoCTA Experience” is by Barbara Wasson. Wasson contributes to knowledge about the pedagogical design of 

network-based learning scenarios, the technological design of the learning environment to support these scenarios, 

and the organisational design for management of such environments by taking a sociocultural perspective on learning 

activity focussing on the interpersonal social interaction in collaborative learning settings. She describes a couple of 

learning scenarios that took place in the DoCTA (Design and use of Collaborative Telelearning Artefacts) project 

together with a discussion and reflection based on the lessons learned in these activities. This paper is an outstanding 

example that illustrates the relationship between the design and use of technology enhanced learning environments 

and how this is tightly intertwined in the institutional, pedagogical and technological aspects of a learning 

environment. 

Jönsson, Mattheos, Svingby & Attström, in their paper entitled ”Dynamic Assessment and the ’Interactive 

Examination’” are interested in assessment and the development of a methodology and technological support to 

assist in the assessment process. A method for supporting university students to carry out self-assessment and to 

compare it with the examiner’s assessment was developed and used in two different evaluation studies. During the 

examination, students assessed their own competence and their self-assessment was matched to the judgment of their 

instructors or to their examination results. Students then received a personal task, which they had to respond to in 

written text. After submitting their response, the students received a document representing the way an “expert” in 

the field chose to deal with the same task. They then had to prepare a “comparison document”, where they identified 

differences between their own and the “expert” answer. The results of this study indicate that the Interactive 

Examination might be a valid methodology for evaluating students’ self-assessment skills, and thus a potential tool 

for assisting the development of certain metacognitive skills in higher education. 

The context for the results reported by Anders Olofsson in his paper entitled “Participation in an Educational Online 

Learning Communities” is the issue of students´ participation in net-based higher education courses. Olofsson claims 

that the understanding of distance education has moved from being a question about transferring knowledge, towards 




others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior 

specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 

1

a question about learning together in an educational online learning community and he raises the issue of which 

pedagogical aspects need to be considered in order to support active student participation in these types of learning 

environments. Using data gathered through semi-structured interviews with 19 trainees on a Swedish network-based 

teacher training program supported by Information and Communication Technologies, Olofsson shows that what 

seems to be required from each trainee, in order to be a member of an educational online learning community, is an 

active participation and an inclusive attitude towards other members within the community. Based on these results, 

the author calls for a pedagogical approach to network-based learning in which the students being-together is taken 

as starting point, and where the pedagogical issues are firmly based on an ethical view of people and education. 

Svensson & Östlund, in their paper entitled “Bridging Design Theory and Distance Educational Practice with 

Techno-Pedagogical Genres” takes up another Scandinavian strength, namely design. The paper argues that design 

concepts should be used to bridge the gap between design theories and distance educational practice. It is also argued 

that genre theory could be instrumental in framing the characteristics of such techno-pedagogical genres in a way 

that constitutes a powerful level of communicating and disseminating new ideas within and across educational 

communities. The framework for techno-pedagogical genres is presented together with three illustrating examples. 

Mikael Wiberg elaborates the concept of learning through networks in his paper entitled “Netlearning and Learning 

through Networks”. This paper is inspired by recent research into the interaction society and the Scandinavian 

tradition in system development that have always highlighted the importance of user-driven processes, users as 

creative social individuals and a perspective on users as creative contributors to both the form, and content of new 

interaction technologies. Wiberg proposes the concept of netlearning as a general label for the traditional use of 

computer-based learning environments as education tools and then, he suggests the concept of learning through 

networks as a challenging concept for addressing user-driven technologies that support social, collaborative and 

creative learning processes in, via, or outside typical educational settings. 

Milrad and Spikol present their efforts in supporting learning using mobile phones in their paper entitled “Anytime, 

Anywhere Learning Supported by Smart Phones: Experiences and Results from the MUSIS project”. This paper 

presents the results of two pilot studies exploring the use of mobile phones in educational settings and the design of 

mobile services to support learning and collaboration in university courses. The results and discussion regarding the 

outcome of these trials are presented together with an explanation of how students experienced the mobile services. 

Issues and problems are discussed with regard to the technology and its use. The authors emphasize the importance 

of usability, institutional support, and tailored educational content in order to increase the potential for successful 

implementation of mobile services in higher education. 

To conclude, Järvelä, Näykki, Laru & Luokkanen present their efforts exploring the possibilities to scaffold 

collaborative learning in higher education with wireless networks and mobile tools in their paper entitled 

“Structuring and Regulating Collaborative Learning in Higher Education with Wireless Networks and Mobile 

Tools”. The authors investigate how pedagogical ideas that are grounded on concepts of collaborative learning, 

including the socially shared origin of cognition, can be supported using mobile phones. Three design experiments 

are presented investigating novel ways to structure and regulate individual and collaborative learning supported by 

smartphones. Based on the results illustrated in this paper, the authors conclude that there is a need to place students 

in various situations in which they can engage in effortful interactions in order to build a shared understanding. 

Wireless networks and mobile tools can provide multiple opportunities for bridging different contents and contexts, 

as well as virtual and face-to-face learning interactions in higher education. 

Having in mind the current stage in development, implementation and assessment of different network-based 

learning approaches, the papers in this special issue provide an illustration of current Scandinavian research efforts in 

this direction. There may be contrasts and similarities between the papers – but this was our intention from the start: 

to open up the discussion about networked-based learning and to look at it from different perspectives. The research 

results presented here provide the chance to look at them beyond the detail of particular studies and consider broader, 

more fundamental questions. With our capacity, as Guest Editors of this special issue, we hope that readers of ET&S 

will value the content and results presented in the different contributions featured in this special issue. 

2

Wasson, B. (2007). Design and Use of Collaborative Network Learning Scenarios: The DoCTA Experience. Educational 

Technology & Society, 10 (4), 3-16. 

Design and Use of Collaborative Network Learning Scenarios: The DoCTA 

Experience 

ABSTRACT 

Barbara Wasson 

InterMedia & Department of Information Science and Media Studies, University of Bergen, Norway // 

Tel: +47 55584120 // Fax: +47 55584188 // barbara.wasson@uib.no 

In the Norwegian DoCTA and DoCTA NSS projects we aimed to bring a theoretical perspective to the design of 

ICT-mediated learning environments that support the sociocultural aspects of human interaction and to evaluate 

their use. By taking a sociocultural perspective on learning activity focussing on the interpersonal social 

interaction in collaborative learning settings we contribute to knowledge about the pedagogical design of 

network based learning scenarios, the technological design of the learning environment to support these learning 

scenarios, and the organisational design for management of such learning environments. Through various 

empirical studies we improved our understanding of the pedagogy and technology of networked learners, and 

increased our understanding of learner activity. This paper reports on the VisArt artefact design scenario and the 

gen-etikk collaborative knowledge building scenario focusing on their design and use. Both scenarios comprised 

co-located and distributed students collaborating over the Internet during a 3-4 week period. 

Keywords 

Collaborative Network Learning; Design and use; Distributed Collaboration; Technology Enhanced Learning 

Introduction 

According to Scandinavian tradition there is a tight relationship between design and use where one is always 

designing for future use situations (Bannon & Bødker, 1991; Wasson, 1998). Human-computer interaction 

researchers, Liam Bannon and Susanne Bødker argue for studying artefacts in use, for studying how they mediate 

use, how they are incorporated into social praxis as the basis for designing future use situations. Design in this 

human activity framework is “a process in which we determine and create the conditions which turn an object into 

an artifact of use. The future use situation is the origin for design, and we design with this in mind … To design with 

the future use activity in mind also means to start out from the present praxis of the future users ((Bannon & Bødker, 

1991, p. 242)”. This raises implications for the design, implementation, use and evaluation of network based 

learning environments. 

In this paper I show how these implications were manifested in the Norwegian DoCTA (Design and use of 

Collaborative Telelearning Artefacts) project (http://www.intermedia.uib.no/docta/hoved2.html) through the 

application of theory to both the pedagogical and technological design, the composition of the design teams and to 

how continuous evaluation, by both participants and researchers feeds into redesign. 

In DoCTA the focus was on the design and use of technological artefacts to support collaborative learning in 

distributed settings (Wasson, Guribye, & Mørch, 2000; Wasson & Ludvigsen, 2003). The objectives of DoCTA 

included: 

• taking a sociocultural perspective on learning activity focusing on the interpersonal social interaction in a 

networked collaborative learning setting 

• contributing to knowledge about the pedagogical design of learning scenarios, the technological design of the 

learning environment to support these learning scenarios, and the organisational design for management of such 

learning environments, including a reflection on teacher and learner roles for collaborative learning in 

distributed settings, and 

• to study and evaluate the social and cultural aspects of collaborative learning in distributed settings 

Through these objectives we aimed to improve our understanding of the pedagogy and technology of networked 

learners, and increase our understanding of learner activity, in order to lead to better design, management and 






3

affordances of networked learning spaces. DoCTA 1 (1998-1999) and DoCTA NSS (2000-2004) were 

interdisciplinary research projects funded by the Network for IT-Research and Competence in Education (ITU) 

which is a measure taken by the Ministry of Education to support ICT and learning in the Norwegian Educational 

system. 

In DoCTA 1 (Wasson, Guribye & Mørch, 2000) we focused on the design and use of technological artefacts to 

support collaborative networked learning aimed at teacher training. The research was not limited to only studying 

these artefacts per se, but included social, cultural, pedagogical and psychological aspects of the entire process in 

which these artefacts are an integral part. This means that we both provided and studied virtual learning 

environments that were deployed to students organised in geographically distributed teams. Various scenarios 

utilising the Internet were used to engage the students in collaborative learning activities. Through participation, the 

students gained experience with not only collaborative learning, but with networked learning through the 

collaborative design of a textual (Scenarios IDEELS and Demeter) or visual artefact (Scenario VisArt). Details of 

these studies can be found in the ITU DoCTA report (Wasson, Guribye & Mørch, 2000). In this paper, the VisArt 

scenario is in focus. 

In DoCTA NSS (Wasson & Ludvigsen, 2003) we investigated, through a design experiment, how the pedagogical 

design of an ICT-mediated collaborative learning environment enables students to learn complex concepts and how 

they can go about discussing these concepts in the broader learning community. Design experiments (Brown, 1992) 

can be seen as intervention in educational practice since the researchers, in collaboration with teachers, try to change 

the way student’s work (Ludvigsen & Mørch, 2003). In our design experiment we intervened in grade 10 natural 

science education by introducing an ICT-mediated collaborative learning scenario in gene technology, gen-etikk, 

where students collaborated both in co-located and distributed settings. The aim was to investigate how the 

pedagogical design of an ICT-mediated collaborative learning environment enables students to talk science and how 

this mediates learning. Augmenting our methodological toolbox with Interaction Analysis (Jordan & Henderson, 

1995) we added studies (Arnseth 2004; Arnseth et al. 2002, 2004; Rysjedal & Wasson, 2005) about the interaction 

between collaborating students and uncovered how the students make their evolving understanding visible to each 

other (Stahl, 2002) and how the artifacts that they use are an integral part of this process. In this paper I focus on 

how the underlying theoretical model of learning had implications for the pedagogical and technological design and 

for our evaluations. 

This paper is organized as follows. The next section looks that the design and use of technology enhanced learning 

environments and presents a conceptual model that illustrates the complexity of this relationship. Then the two 

scenarios, VisArt and gen-etikk are presented, respectively, with a focus on their design and use. The paper 

concludes with a general discussion. 

Design and Use of Technology Enhanced Learning Environments 

From a socio-cultural perspective on learning the notion of activity is seen as the basic concept for design and 

analysis (Ludvigsen & Mørch, 2003). This view, together with the Bannon & Bødker view of designing for future 

use situations, implies that when we look at a technology rich learning environment we need to look at activity from 

both a design and use perspective. Figure 1 illustrates this tight relationship. The TEL design addresses institutional 

(or organisational), pedagogical and technological aspects of the learning environment and the activity that emerges 

from implementing the design, that is the TEL environment in use, can been evaluated from an institutional, 

pedagogical or technological perspective. This means that when designing a learning scenario, the pedagogical and 

technological design is important and it is tightly entwined in an institutional context. It also implies that 

understanding the use is a complex relationship between institutional, pedagogical and technological perspectives. 

The institutional aspects often set the constraints for a learning scenario and are the aspect on which the designer has 

the least impact. 

The tight interaction between design and use, as illustrated in figure 1, shows that the design of a technology 

enhanced learning scenario requires the design of the institutional (or organizational), the pedagogical and the 

technological aspects. A theoretical perspective can influence the design (as illustrated in the section on the gen-etikk 

scenario). Implementation of the design can entail aspects such as tailoring or developing technology, intervention in 

existing practice (e.g., in a classroom), or pedagogical redesign of existing learning activities. Understanding the use 

4

of the technology enhanced learning environment can be taken from an institutional, a pedagogical or a technological 

perspective (example of this will be shown later in the description of the VisArt scenario). In order to understand 

the use, the evaluation, influenced by ones theoretical perspective, can take the form of experiments, field trials, 

ethnographies, etc., or some combination of these. The results of the evaluation inform the (re)design process. 

VisArt 

Figure 1. Design and Use of Technology Enhanced Learning Environments 

In DoCTA 1, the design and use of the VisArt scenario involved a networked learning environment situated in a 

higher education setting where the students participated in the collaborative design of a visual artefact for use in 

teaching. There are a number of research areas that have influenced DoCTA 1 (see Wasson, Guribye & Mørch 

(2000) for details). The two most significant are the conceptual framework offered by sociocultural perspectives 

(Wertsch, del Río & Alvarez, 1995) and the fields of computer support for collaborative learning (CSCL), in 

particular Salomon’s (1992) work on genuine interdependence. We also take inspiration and guidance from CSCW 

theory, in particular ideas on awareness (Dourish & Bellotti, 1992; Gutwin et al., 1995) and coordination science 

(Malone & Crowston, 1994). These theoretical perspectives have influenced our choice of groupware tool, the 

design of the VisArt scenario and its collaborative activity, and the design of our evaluation studies. 

Design of VisArt 

The design of the VisArt activity took place over four months. The design team comprised the DoCTA researchers 

and the instructors for the participating courses. We developed the student´s design activity, specified the 

technological environment, coordinated dates for the activity, designed the training, help and assistance for the 

deployment, and designed our evaluations. 

Figure 2 identifies the institutional, technological and pedagogical aspects of the VisArt scenario. The institutional 

aspects encompass students taking different courses at three geographically dispersed Norwegian higher educational 

institutions, the University of Bergen, Nord-Trøndelag College, and Stord/Haugesund College. The learner’s 

participating in the scenario had different backgrounds, ranged in age from 23 to 68 years and many had family 

responsibilities and full time jobs. The University of Bergen students were taking a graduate course in pedagogical 

information science and were learning about computer support for collaborative learning. They were a blend of 

pedagogical information science graduate students (with a teacher’s background) and information science graduate 

students (with a social science background). The students at Stord/Haugesund College were senior undergraduate 

students training to be teachers who were taking a distance learning course on pedagogical information science that 

5

included a unit on collaborative learning. The Nord-Trøndelag students were taking an undergraduate introduction 

course on the uses of technology in learning and had a choice to participate in the scenario to learn about networked 

learning. 

Figure 2. Design of VisArt 

There were a number of research areas that influenced the pedagogical design. The two most significant are the 

conceptual framework offered by sociocultural perspectives and the field of computer support for collaborative 

learning, in particular Salomon’s (1992) work on genuine interdependence. Thus we designed a collaborative 

learning activity where the students not only participated in teams collaborating to design a learning activity, but they 

had to reflect on their participation. The students were informed that in the VisArt activity they would be part of a 

team of 3 students comprised of 1 student from each institution. Thus each participant had complementary 

backgrounds; this is in alignment with Salomon´s (1992) idea that collaboration is only successful when there is a 

genuine interdependence between the collaborators. There were no opportunities for the team members to meet faceto-face. 

The team was to: 

• Organise a collaborative team effort thinking of Salomon’s definition genuine interdependence: 1) sharing 

information 2) division of labour 3) joint thinking 

• Carry out the Design Activity in TeamWave Workplace (TW) 

• the room you chose to design should enable the students to know more about a concept, a procedure, a 

theory, a process, etc. 

• with the aid of help pages, assistance (from a course assistant), Help room in TW 

• Produce 2 items: 

• A document of your pedagogical decisions (e.g., who is the room intended for, the content, etc….) 

• A TW room for teaching/learning 

In addition to participating in the VisArt activity, the UiB students were to produce an individual report on their 

experience with VisArt that contained: 

• an introduction to CSCL and networked learning 

• a description of the design activity, including the tools provided and used 

• a presentation of their team’s room and the pedagogical decisions made 

• a discussion of how the team met Salomon's requirements for genuine interdependence and whether or not 

TeamWave Workplace supported activities resulting from attempts at meeting Salomon’s requirements 

• a discussion of Gutwin et al.’s awareness concept and what it means in conjunction with your distributed 

collaboration through TW 

• their general reaction to collaborative telelearning (as you experienced it) including: a reflection on the team’s 

work, the process of carrying out the assignment, general comments about the entire assignment, your reaction 

to TW 

6

The technological aspects of the VisArt scenario comprised three tools, TeamWave Workplace, their own email 

system and a web browser. TeamWave Workplace (TW), a groupware tool developed at the GroupLab at the 

University of Calgary, was used as the main information and communication technology. One important distinction 

between TW and other real-time groupware systems is that TW is based on the metaphor of a place (i.e., a room), 

while most others are based on the metaphor of a meeting. A major strength of TW is that it provided a wellintegrated 

set of varied collaboration tools with a good blend of real-time (synchronous) and asynchronous 

communication tools that enable anytime team collaboration. Furthermore, TW augments both existing user 

interaction tools such as email, newsgroups and conferencing, and existing conventional applications such as word 

processors and spreadsheets. Another strength of TW’s integrated approach is that spontaneous as well as preplanned 

intra-team interactions are supported. From a CSCL theoretical perspective, TW enables collaboration and 

supports genuine interdependence between team members. Team members are able to share information, meanings, 

thoughts, conceptions and conclusions through their choice of operational tool objects. From a sociocultural 

perspective it can be said that these thinking tools (signs) in turn facilitate both team-mate knowledge construction 

and collective growth. Furthermore, the thinking tools provide a means for thoughts to be examined changed and 

elaborated upon by fellow team members. 

TeamWave was used for the IDEELS scenario in DoCTA 1 so we had already determined how to use the tool with 

teams and attempted to design a new tool to include in TW. This prove impossible for reasons beyond our control, 

thus we changed the design activity to utilise the exiting tools, highlighting the relationship between the 

technological and pedagogical aspects. Figure 3 shows the Classroom room that was the starting room for the 

students. The tools Calendar, To Do List, Doors (to other rooms), Address Book, URL links, Chat, Awareness Lists 

and a File (Visart oppgaver) can be seen. 

Implementation of VisArt 

The VisArt activity was deployed for 1 month beginning on February 25 th and ending on March 26 th . The students 

were given 5 days to download TW, test their accounts and team email addresses. A training-phase (in TW) lasted 

for ten days, and the main goal for this activity was for the students to get to know TW, to get to know the other 

members of their team, and to also give them some ideas on how to work and collaborate in TW. A three-week 

design activity followed. The students collaborated to create a learning room in TW. There were 11 teams and the 

topics they chose to design learning rooms for included: endanger species, gothic art, publishing on the internet, 

triangles, the big bang, travelling in Denmark, renewable energy sources, between the world wars, polar bears, and 

astronomy. Figure 4 shows the 2 rooms that were designed for learning about polar bears. 

Evaluation of use of VisArt 

The evaluation of the VisArt scenario was carried out on several levels and from several perspectives. The 

theoretical, or conceptual approach, to the evaluation of VisArt was rooted in a sociocultural perspective that 

emphasises an understanding of language, culture and other aspects of social setting and focus on the use situation. 

Ethnographic studies, favouring naturalistic and qualitative research methods were employed. In addition to the 

student’s own theoretical reflections (which are important in a sociocultural perspective), VisArt was evaluated as 

part of eight Master’s theses, including two activity theory studies of how students (Andreassen, 2000), instructors 

and facilitators (Wake, 2002) organise their work. In addition, these ethnographic flavoured studies were augmented 

with a usability study of TW (Rysjedal, 2000), looking at the efficiency of TW from a qualitative perspective using 

the data logs generated by TW (Meistad 2000, Meistad & Wasson, 2000), performing a formative evaluation of how 

to support collaborative design activities, seeing how TW supports coordination, how to design training and 

assistance in a collaborative telelearning setting (Underhaug, 2001). Some of the results of these studies are 

presented below. 

In general, the students were very satisfied with TW. As one student writes in his reflection over TW 

“An important side with TeamWave is that one can work both asynchronously and synchronously. … For 

example one can use the shared whiteboard synchronously when the users are online at the same time and 

write on it together, but it is also possible to use the whiteboard asynchronously when the different users log 

7

on at different times and work individually on tasks on the whiteboard. …That it supports both forms of 

work makes the program package flexible and accessible at all times.” 

Figure 3. Screenshot of the Classroom in TeamWave Workplace 

Figure 4. Screenshots of rooms for learning about Polar Bears 

Several students wrote that the successful use of TW was not just tied to the ease of use, rather, that it is used in an 

activity that meets Salomon’s requirements. As one student succinctly put it 

“I think that a requirement for successful use of it [TW] is that the participants are motivated and have 

mindful engagement and that the tool [TW] is used for something meaningful.” 

The majority of the groups had a heterogeneous makeup with the group members having different backgrounds. As 

one group said, this meant that they had different preconceptions and different experiences with collaboration. They 

said, that 

8

“according to Salomon it is exactly these differences that makes collaboration work…to use each others 

competence and pull something useful of these competencies through collaboration.” 

From a sociocultural research perspective the student’s own reflections are a very important part of evaluation and as 

illustrated in the previous excerpts they demonstrated an ability to reflect theoretically on practice. The research 

reports they submitted in the course held comments and reflections that were both thoughtful and insightful and will 

lead to improvements in future versions of the scenario. 

From an ethnographic flavoured study looking at how students organized their work in VisArt, Andreassen (2001) 

used different qualitative data gathering techniques from a variety of data sources, ranging from electronically 

collected TW chat logs to transcribed informal interviews. The data analysis dealt with aspects like co-ordination, 

communication mode, division of labour, and feedback. During the entire scenario the students met regularly in TW, 

and co-ordinated their actions by using TW or email. Sometimes both TW and email were used as a means of coordination, 

providing a form of double communication. This form of communication disappeared as the scenario 

went on, maybe as a result of the establishment of regular meetings and patterns for collaboration. The task decision 

marked a noticeable line of demarcation in the communication mode. Before deciding on the task the rate of 

synchronous meetings and communication were higher than after the decision had been made. The asynchronous 

nature of the post decision work, may have its root both in the fact that the need for synchronous meetings were 

diminished, and that each student was assigned her/his own area of responsibility, contributing to a co-operative, 

rather than a collaborative form of work. In spite of a mutual agreement on providing feedback on each other’s work, 

this hardly ever occurred. Time pressure and a feeling that one had to concentrate on what oneself was doing, are 

probably the main reasons for the lack of feedback. 

Collaboration patterns define sequences of interaction among members of a team (such as students). In the VisArt 

scenario we have searched for collaboration patterns by analysing interaction data from data logs, videotapes, 

observations, and interviews both between students and between students and facilitators (instructors and assistants). 

We have identified several instances that we believe can be characterised as collaboration patterns (Wasson & 

Mørch, 2000): 

Adaptation: This pattern describes how students gradually adapted to each other’s practices when working together 

to solve a common problem. 

Coordinated desynchronisation: This pattern describes how coordination of activities between team members 

changes after they have identified a common goal. 

Constructive commenting: This pattern describes commenting behaviour. Comments that are neutral (e.g., just to the 

point) are perceived to be less useful than comments that are also constructive (e.g., suggesting what to do next) or 

supportive (e.g., encouraging). 

Informal language: This pattern describes how interaction often starts in a formalistic style and gradually becomes 

more informal as team members get to know each other. Frequent use of slang words or dialects local to the 

community working together is common in instances of this pattern. 

Collaboration patterns are useful for the re(design) of the learning scenario. For example, in the initial phases of a 

collaboration effort, a sort of double communication might occur; more than one tool is used to inform other team 

members about a changed meeting time. This type of adaptation pattern was observed in VisArt and lead to 

inefficiencies. This sort of communication may be reduced or disappear with improved technical understanding or 

changed work coordination over time, but might be avoided with sufficient training and examples on how different 

tools can be used for coordination purposes. 

Our general findings in DoCTA 1 include: 

• the ease of use of the many collaborative tools tells us that the technological problems are no longer the prime 

issue in CSCL design 

• the main issues are related to the broader institutional contexts in which the tools are designed and used 

• coordination issues remain a challenge for collaboration with distributed learners 

9

• the numerous evaluation studies that we have carried out not only contribute to our understanding of the social 

and cultural aspects of collaborative networked learning environments, but equally important, they have also 

addressed methodological issues related to studying online environments 

• the reflections of the students played an important part in our understanding of the learning activity and feed into 

future designs 

Gen-etikk 

In DoCTA NSS we intervened in grade 10 natural science education by introducing an ICT-mediated collaborative 

learning scenario in gene technology, gen-etikk. In gen-etikk a cross curriculum scenario of natural science, religion 

& ethics (KRL) and Norwegian was developed collaboratively between the researchers and teachers and the learning 

goals related to the biological, ethical and societal aspects of gene technology. The pedagogical approach was 

progressive inquiry learning (Muukkonen, Hakkarainen, & Lakkala, 1999) and a web-based groupware system, 

FLE3, that supports this model was used as the main learning technology. Figure 5 illustrates that the progressive 

inquiry learning theoretical model was operationalised in both the pedagogical and technological design of gen-etikk. 

Students in two classes collaborated in both co-located (within groups in a class) and distributed (between groups in 

two different Norwegian cities) settings to share and discuss ideas and arguments around scientific and ethical 

questions related to gene technology. In this section we elaborate on the design rationale behind the scenario by 

detailing the pedagogical approach and the didactic design and then introduce the technological environment and 

describe the deployment of the scenario. 

Design of Gen-etikk 

In the design processes a key aspects in this kind of design experiment is both to adapt to the schools everyday 

practice, and on the other side, challenge and extend these practices. Progressive inquiry learning is an approach to 

collaborative knowledge building where students engage in a research-like process to gain understanding of a 

knowledge domain by generating their own problems, proposing tentative hypotheses and searching for deepening 

knowledge collaboratively. As a starting point for progressive inquiry learning, a context and the goal for a study 

project needs to be established in order for the students to understand why the topic is worthwhile investigating. 

Then the instructor or the students present their research problems/questions that define the directions where the 

inquiry goes. As the inquiry cycle proceeds, more refined questions will emerge. Focusing on the research problems, 

the students construct their working theories, hypotheses, and interpretations based on their background knowledge 

and their research. Then the students assess strengths and weaknesses of different explanations and identify 

contradictions and gaps of knowledge. To refine the explanation, fill in the knowledge gaps and provide deeper 

explanation, the students have to do research and acquire new information on the related topics, which may result in 

new working theories. In so doing, the students move step by step toward building up knowledge to answer the initial 

question. The role of the teachers is to be a facilitator for the students. The teachers can stimulate self-regulation by 

the students by giving comments and advice, both within the classroom and in the online environment. 

Figure 5. Operationalisation of Progessive Inquiry Learning 

10

The pedagogical design was inspired by the progressive inquiry approach to knowledge building. Animated by a 

trigger video (we edited a Norwegian National Broadcasting Corporation (NRK) documentary on gene technology to 

4 5-minute segments, each presenting a different theme within genetic technology) to set the context and supported 

by the structure and resources in the learning environment, the students themselves will identify problems on which 

to work, decide where they wanted to search for information, participate in inquiry learning cycles and create 

newspaper articles. We developed a set of activities with instructions that included assignments related to the inquiry 

learning cycle (e.g., generate scientific and ethical questions about gene technology; engage in inquiry about selected 

questions, compose scientific explanations, etc.) and products related to expressions of what they have learned 

(scientific and ethical questions, science questions for use on a test, write individual and collaborate texts on opinions 

about an argument or a discussion about a scientific or ethical question to be published in the national school 

newspaper). 

For the technological design, support for gen-etikk was given through a web portal that was designed in order to 

provide the students with a shared online space (see figure 6). From this portal the students had access to various 

learning resources, collaboration tools, and a tool for Internet publishing called Skoleavisa (an online newspaper 

generator available for all schools in Norway). Among the learning resources they could find an online text book 

(previously written by 2 of the DoCTA researchers), a Norwegian encyclopaedia, animations, a special search engine 

Atekst (searches Norwegian newspaper archives) and some selected links to external resources on the Internet. 

The main tool for collaboration was Future Learning Environment 3, FLE3 (http://fle3.uiah.fi). FLE3 is designed to 

support collaborative knowledge building and progressive inquiry learning (Muukkonen et al., 1999) and comprises 

several modules. The Web Top provides each group with a place where they can store and share digital material with 

other groups. An automatically generated message that tells what has happened since the last time they visited FLE3 

also appears here. The Knowledge Building module is considered to be the scaffolding module for progressive 

inquiry and it can be seen as a semi-structured communication interface (Dillenbourg, 2002). It is a shared database 

where the students can publish problem statements or research questions, and engage in knowledge building 

dialogues around these problems by posting their messages to the common workspace according to predefined 

categories which structure the dialogue. These categories are defined to reflect the different phases in the progressive 

inquiry process, thus they operationalise the theory in the tool. These included: Question, Our explanation, Scientific 

explanation, Summary, Comment and Process Comment. We added a digital assistant to FLE3 (Chen & Wasson, 

2003) to support both the students and teachers in monitoring what happened inside FLE3 (Dragsnes, Chen, & 

Baggetun, 2002). In addition to FLE3, a combined chat and mind mapping tool (Dragsnes, 2003) was developed and 

made available for the students to add support for synchronous communication. 

Implementation of Gen-etikk 

Gen-etikk took place over 31 hours during the three last weeks of September 2002, and involved two grade 10 

classes, one from Bergen (24 students) and one from Oslo (27 students). Five of the 31 hours were concurrent (i.e., 

both classes worked on gene-tikk at the same time) and synchronous communication was possible. The scenario 

began with each class viewing the trigger video on genetic technology. Then the students brainstormed about 

questions related to genetic technology. This brainstorming session generated a long list of questions from the two 

classes, and the teachers used these questions in order to make one single list of questions with 12 scientific 

questions and 12 ethical questions about genetics. This list of questions was published on the web portal. 

The two classes were then divided into local groups with 3 or 4 members, and each of the local groups in Bergen 

was connected to a local group in Oslo to form a composed group. The scenario had two phases, and in the first 

phase the composed groups discussed the list of questions and decided on three scientific questions to work on. 

These questions were posted as problem-statements in FLE3 before they started to search for and discuss information 

around their questions. Whenever they found something relevant, they could post it as a note in the knowledge 

building module in FLE3. After having explored the questions for about a week the students should use the 

information they had gathered in order to write at least two different articles about genetics. These articles were 

published in Skoleavisa, the online newspaper generator. 

In the second phase of the scenario the focus was turned to the ethical aspects of gene technology. The list of 

questions was revisited, and this time the composed groups should decide on 3 ethical questions on which they 

11

wanted to work. The same inquiry process was repeated in this phase, with about one week of inquiry of questions 

before publishing articles in Skoleavisa. It was believed that focusing on scientific aspects before they turned to the 

ethical aspects would increase the students’ abilities to argue on their ethical viewpoints. By the end of the project 

every group had contributed and 60 articles were published in the online newspaper. 

Evaluation of the use of gen-etikk 

Figure 6. The Web Portal for gen-etikk 

There have been a number of empirical studies carried out on the DoCTA NSS data. The design of a learning 

environment needs to account for institutional, technological and pedagogical aspects at different levels. Three of the 

evaluations that take an institutional perspective on learning are summarised in this section. An institutional 

perspective takes student actions and activities as a staring point, not the goals in the curriculum or some scientific 

template. Diversity, multiple voices, the actors’ different goal and intensions, and the institutional history are some 

of the aspects that constitute a specific practice. These aspects create a basis for understanding how students act in 

specific situations. 

Rysjedal and Baggetun (2003) discuss issues related to infrastructure and design of learning environments. The 

established infrastructure in an institution creates both constraints and affordances for how new technology can be 

integrated. In the design of a learning environment that should work across institutional boundaries it is important to 

take the local infrastructures into consideration. Rysjedal and Baggetun take a broad perspective on infrastructure, 

and thereby make an important bridge between technological and social perspectives on how new technology can be 

introduced into social systems. They discuss how the design of a learning environment needs to take technological, 

organisational and pedagogical aspects into consideration. 

In Arnseth, Ludvigsen, Guribye and Wasson (2002) and Arnseth (2004) describe how rhetorical aspects of human 

talk and discourse become important if we want to understand how students co-construct knowledge in schools. 

Their empirical analyses show very clearly that students make specific interpretation of the task to which they are 

exposed and to how the institution actually works. The authors argue that knowledge building as a metaphor as used 

in some of the literature seems too be too rationalistic. In a similar vein, Ludvigsen and Mørch (2003) criticize the 

progressive inquiry model proposed by Mukkonenen et al. (1999) because it has a too distinct focus on the 

12

conceptual artefacts developed by the students, and that the progressive inquiry model is privileged as the analytic 

staring point. 

A selection of the lessons learned from DoCTA NSS as reported in Wasson & Ludvigsen (2003) include: 

• Our major finding is that too few students use higher order skills as part of their learning activities. This 

confirms the findings reported in many international studies. Students and teachers have a tendency to place 

more importance on solving the task than on the domain concepts to be learned. Students need to employ 

higher order skills when dealing with knowledge building in complex and conceptually-oriented environments 

in order to go beyond fact finding. 

• The teacher is extremely important in supporting, stimulating and motivating the students to integrate previous 

knowledge with the new knowledge they are learning through the gen-etikk tasks. 

• We find the same tendency as shown in the PISA study (Lie et al. ) that students do not have good enough 

learning strategies. When meeting new ICT-supported learning situations, students need time and training in 

their integrated use before their learning strategies become effective. 

• Prompting categories triggered some of the students to a more critical and analytic stance towards the learning 

resources and how they reason about ethical issues in the domain of gene technology. 

• The students that engage themselves in the task at a deep level show evidence of the necessary skills needed to 

critically examine the relationship between information and the argumentation that is part of the problem solving 

process. 

• The design, which includes small group collaboration, creates increased motivation and curiosity. 

• When schools work together to create a distributed environment where students solve tasks together, the 

management of the time schedules of the two schools needs to adjusted – or the school needs to have a flexible 

time schedule. Practical arrangements create tensions and problems with the coordination between the schools. 

• Students have little problem in the practical use of ICT-tools as long as the tools and network function as they 

should. 

• Several types of digital resources were created to support the development of the ability to integrate information 

from different resources as part of knowledge construction. This is one important aspect in designing for the 

cultivation of higher order skills. 

Discussion and Conclusions 

This paper has attempted to illustrate the relationship between the design and use of technology enhanced learning 

environments and to illustrate how this is tightly intertwined in the institutional, pedagogical and technological 

aspects of a learning environment. Furthermore, the view of design and use is heavily influenced by a sociocultural 

perspective on learning that views activity as central to both design and analysis. The DoCTA 1 and DoCTA NSS 

scenarios have been described in a way that highlights the pedagogical, technological and institutional design aspects 

and how the evaluation studies have looked at the use, or activity as it emerges. Several general observations have 

been presented for each of the projects. 

Having the opportunity to work with networked learning over a number of years has had its advantages. Early on we 

learned that it rarely was a problem with introducing a new technology to our students, and this was true for both 15 

year olds and university students. What we encountered, in both groups, however, was their desire to use the 

technological tools that they already used in their daily lives in our scenarios and we tried to accommodate this when 

possible. For example, in VisArt, we incorporated their own email into the scenario and adjusted our data collection 

to collect this email as well. In gen-etikk we found that the teenagers wanted to use IRC for having contact between 

the distributed groups, so we incorporated this as well. In later projects we found that they used their mobile phones 

for coordination and collaboration. 

We also learned a lot about evaluation of networked learning, the main one is that there is no recipe for how to 

evaluate and carry out analysis of networked learning. In DoCTA 1 we learned that there are many ways from which 

a technology enhanced learning scenario can be viewed and that only one view will not tell the story. For example, a 

technology that in incorporated into a pedagogical design may prove to be the wrong tool, or may have problems due 

to the institutions infrastructure. It is not as simple as saying “the technology did not support learning”. Maybe the 

usability of the tool is poor, or it did not fit the task for which it was chosen. Thus human computer interaction 

13

usability studies have a place in the evaluation repertoire, but are only part. In DoCTA NSS we paid attention to the 

unit of analysis. For example, we tried to build on state-of-the-art knowledge in order to design gen-etikk. The 

pedagogical and technological designs form conditions for how and what students could learn. The designed 

environment, however, is only one important aspect of what we need to understand. As Wasson & Ludvigsen (2003) 

have argued, learning and knowledge building are always part of an institutional arrangement, and we need to take 

this as a starting point. The socio-cultural perspective gives us possibilities to understand how higher order skills can 

be developed. By having insight into student’s learning trajectories, the kind of talk in which they are engaged, and 

in how the division of labor is distributed between the students and teachers, we begin to understand how the 

cultivation of higher order skills becomes part of institutionalized activities; otherwise it will be serendipitous. Only 

by looking at the chosen technology in relation to these other aspects can we say anything about how it supports 

learning. Thus we can arguer that institutional, technological and pedagogical aspects need to be treated as a unit of 

analysis in the design processes. Furthermore, the theoretical underpinnings, technological artefacts and the 

evaluation of use need to mutually inform each other. As illustrated in both scenarios, the designer´s theoretical 

perspective on learning influences the design of the pedagogy and can also, as in VisArt, be embedded in the 

technological tools. The theoretical perspective also has implications for the methodology and methods of analysis. 

When looking to the future the challenges for networked learning are many, but many of them exciting. Given 

increased mobility in work situations and in society in general, mobile and wireless technologies are becoming vital 

artefacts in all aspects of our lives. New technological advancements make collaboration across devices (e.g., mobile 

phones, PCs, PDAs, PocketPCs, etc.) and networks (e.g., GSM, GPRS, 3G, LANs, WLANs, WMANs) possible and 

opens for learning areans. For example third generation (3G) handsets allow users of 3G services to view and record 

video content and television in addition to WAP-functionalities such as reading e-mail or surfing on the Web. 

Furthermore, software is becoming available or accessible from a variety of devices (on e.g. PDAs, Pocket PCs, 

mobile phones). These new technological advancements place high demands new digital and mobile literacies among 

learners. Consequently, in-depth and structured knowledge on how wireless and mobile technologies impact human 

actions in learning is limited and raises a plethora of specific questions about how these technologies change 

collaborating institutions and their pedagogy. Recent studies are beginning to make headway into understanding 

these new conditions for learning (the collection in Arnedillo-Sánchez, Sharples & Vavoula, 2007; Baggetun & 

Wasson, 2006; Kukulska-Hulme , 2007; Milrad & Jackskon, 2007; Thackara, 2005). 

Finally, I believe that we are dealing with a new type of student. Technology savvy students, who are used to 

configuring their own virtual world, organise their own interactions with peers, instructors and the world beyond. 

Games such as World of Warcraft and Social tools such as flikr, del.icious, YouTube, Facebook, blogging, WireHog, 

Groove are in the everyday repertoire of our current and future students and I ask, how are we to design learning 

environments for youth who have the world at their fingertips and how are we to capture their attention in order to 

learn something that we think is important. 

Acknowledgments 

DoCTA was funded by The Norwegian Ministry of Education under their Information Technology in Education 

(ITU) programme. I thank the teachers/instructors and students from the participating schools/colleges. Finally, I 

commend all the project participants for creating such an exciting and stimulating project environment. 

References 

Andreassen, E. F. (2000). Evaluating how students organise their work in a collaborative telelearning scenario: An 

Activity Theoretical Perspective, Masters dissertation, Dept of Information Science, University of Bergen, Norway. 

Arnseth, H. C. (2004). Discourse and artefacts in learning to argue: Analysing the practical management of 

computer supported collaborative learning, Ph.D. dissertation, University of Oslo, Norway. 

Arnseth, H. C., Ludvigsen, S., Guribye, F., & Wasson, B. (2002). From Categories of Knowledge Building to 

Trajectories of Participation. Analysing the Social and Rhetorical Organisation of Collaborative Learning. Paper 

presented at the ISCRAT 2002, June 18-22, 2002, Amsterdam. 

14

Arnseth, H.C., Ludvigsen, S., Mørch, A., & Wasson, B. (2004). Managing Intersubjectivity in Distributed 

Collaboration. Psychnology, 2 (2), 189-204. 

Bannon, L. J., & Bødker, S. (1991). Beyond the interface. Encountering artifacts in use. In J. M. Carroll (Ed.), 

Designing interaction: Psychology at the human-computer interface, Cambridge, UK: Cambridge University Press, 

227-253. 

Brown, A. L. (1992). Design Experiments: Theoretical and methodological challenges in creating complex 

interventions in classroom settings. The Journal of the Learning Sciences, 2 (2), 141-178. 

Chen, W., & Wasson, B. (2003). Coordinating Collaborative Knowledge Building. International Journal of 

Computer and Applications, 25 (1), 1-10. 

Dourish, P., & Bellotti, V. (1992). Awareness and Coordination in Shared Workspaces. Proceedings of the 

Conference on Computer-Supported Cooperative Work, 107-114, retrieved October 15, 2007, from 

http://www.ics.uci.edu/~jpd/publications/1992/cscw92-awareness.pdf. 

Dragsnes, S. (2003). Development of a synchronous, distributed and agent-supported framework : exemplified by a 

mind map application, Masters dissertation, Department of Information Science, University of Bergen, Norway. 

Dragsnes, S., Chen, W., & Baggetun, R. (2002). A Design approach for Agents in Distributed Work and Learning 

Environments. Paper presented at the International Conference on Computers in Education, December 3-6, 2002, 

Auckland, New Zealand. 

Guribye, F. (1999). Evaluating a collaborative telelearning scenario: A sociocultural perspective, Masters 

dissertation, Department of Information Science, University of Bergen, Norway. 

Guribye, F., & Wasson, B. (2002). The ethnography of distributed collaborative learning. In G. Stahl (Ed.), 

Proceedings of CSCL 2002, Boulder, CO, USA, 637-638, retrieved October 15, 2007, from 

http://www.cis.drexel.edu/faculty/gerry/cscl/cscl2002proceedings.pdf. 

Gutwin, C., Stark, G., & Greenberg, S. (1995). Support for Workspace Awareness in Educational Groupware. 

Proceedings of the ACM Conference on Computer Supported Collaborative Learning, Hillsdale NJ: Lawrence 

Erlbaum, 147-156. 

Jordan, B., & Henderson, A. (1995). Interaction Analysis: Foundations and Practice. Journal of the Learning 

Sciences, 4 (1), 39-103. 

Kukulska-Hulme, A. (2007) Mobile Usability in Educational Contexts: What have we learnt? The International 

Review of Research in Open and Distance Learning, 8 (2), 1-16. 

Ludvigsen, S., & Mørch, A. (2003). Categorisation in knowledge building. In B. Wasson, S. Ludvigsen & U. Hoppe 

(Eds.), Proceedings of the 6th International Conference on Computer Support for Collaborative Learning, 

Dordrecht: Kluwer, 67-76. 

Malone, T., & Crowston, K. (1994). The Interdisciplinary study of coordination. ACM Computing Surveys, 26 (1), 

87-119. 

Meistad, Ø. (2000). Collaborative telelearning: Using log-files to identify collaboration patterns, Masters 


Meistad, Ø., & Wasson, B. (2000). Using server-logs to support collaborative telelearning research. In J. Bourdeau & 

R. Heller (Eds.), Proceedings of Educational Multimedia & Educational Telecom ’2000, Charlottesville, VA: 

AACE, 679-683. 

15

Milrad, M., & Jackskon, M. (2007). Designing and Implementing Educational Mobile Services in University 

Classrooms Using Smart Phones and Cellular Networks. International Journal of Engineering Education, 23 (4). 

Muukkonen, H., Hakkarainen, K., & Lakkala, M. (1999). Collaborative Technology for Facilitating Progressive 

Inquiry: Future Learning Environment Tools. Paper presented at the International Conference on Computer Support 

for Collaborative Learning, 12–15 December 1999, Palo Alto, CA, USA. 

Rysjedal, K.H. (2000). Teamwave Workplace in Use: A useability study, Masters dissertation, Department of 

Information Science, University of Bergen, Norway. 

Rysjedal, K., & Wasson, B. (2005). Local and distributed interaction in a collaborative knowledge building scenario. 

Paper presented at the International Conference on Computer Support for Collaborative Learning, 30 May - 4 June 

2005, Taipei, Taiwan. 

Salomon, G. (1992). What does the design of effective CSCL require and how do we study its effects? SIGCUE 

Outlook, 21 (3), 62-68. 

Stahl, G. (2002). The complexity of a collaborative interaction. Paper presented at the International Conference of 

the Learning Sciences (ICLS 2002), October 23-26, 2002, Seattle, WA, USA. 

Thackara, J. (2005). In the Bubble: designing in a complex world, Cambridge, MA: The MIT Press. 

Underhaug, H. (2001). Facilitating Training and Assistance in a Collaborative Telelearning Scenario, Masters 


Wake, J. (2002). How instructors organise their work in a collaborative telelearning scenario, Masters dissertation, 

Department of Information Science, University of Bergen, Norway. 

Wasson, B. (1998). Identifying Coordination Agents for Collaborative Telelearning. International Journal of 

Artificial Intelligence in Education, 9, 275-299. 

Wasson, B., Guribye, F., & Mørch, A. (2000). Project DoCTA: Design and use of Collaborative Telelearning 

Artefacts, Oslo: Unipub forlag. 

Wasson, B., & Ludvigsen, S. (2003). Designing for knowledge building, Oslo: Unipub forlag. 

Wasson, B., & Mørch, A.I. (2000). Identifying collaboration patterns in collaborative telelearning scenarios. Journal 

of Educational Technology & Society, 3 (3), 237-248. 

Wertsch, J. V., del Río, P., & Alvarez, A. (1995). Sociocultural studies: history, action and mediation. In J.V. 

Wertsch, P. del Río & A. Alvarez. (Eds.), Sociocultural Studies of Mind, Cambridge: Cambridge University Press, 1- 

34. 

16

Jönsson, A., Mattheos, N., Svingby, G., & Attström, R. (2007). Dynamic Assessment and the ”Interactive 

Examination”. Educational Technology & Society, 10 (4), 17-27. 

Dynamic Assessment and the “Interactive Examination” 

Anders Jönsson, Nikos Mattheos, Gunilla Svingby and Rolf Attström 

Malmö University, SE-205 06 MALMÖ, Sweden 

anders.jonsson@mah.se // nikolaos.mattheos@mah.se // gunilla.svingby@mah.se // rolf.attstrom@mah.se 

ABSTRACT 

To assess own actions and define individual learning needs is fundamental for professional development. The 

development of self-assessment skills requires practice and feedback during the course of studies. The 

“Interactive Examination” is a methodology aiming to assist students developing their self-assessment skills. 

The present study describes the methodology and presents the results from a multicentre evaluation study at the 

Faculty of Odontology (OD) and School of Teacher Education (LUT) at Malmö University, Sweden. During the 

examination, students assessed their own competence and their self-assessments were matched to the judgement 

of their instructors (OD) or to their examination results (LUT). Students then received a personal task, which 

they had to respond to in written text. After submitting their response, the students received a document 

representing the way an “expert” in the field chose to deal with the same task. They then had to prepare a 

“comparison document”, where they identified differences between their own and the “expert” answer. Results 

showed that students appreciated the examination in both institutions. There was a somewhat different pattern of 

self-assessment in the two centres, and the qualitative analysis of students’ comparison documents also revealed 

some interesting institutional differences. 

Keywords 

Assessment, Self-assessment, Oral health education, Teacher education 

Introduction 

One of the major challenges for profession-directed higher education today, is not only to equip students with 

knowledge and skills, but also to help them develop into independent learners, able to cope with an ever increasing 

amount of information and learning needs. The basis of this process lies in the individual’s ability to continuously 

assess his or her actions and define individual learning needs accordingly. Research has shown that the ability to 

assess ourselves, especially within professional settings, is not a quality we are born with, but rather a metacognitive 

skill which can be learned, improved and excelled (Brown et al., 1997). It is also shown that not all professionals 

have developed this ability to a satisfactory degree and might consequently be unable to identify shortcomings in 

their own professional competence (Hays et al., 2002; Ngan & Amini, 1998; Reisine, 1996). But if professional 

education is supposed to foster reflecting and self-assessing practitioners, the students must be given the opportunity 

to practice these skills (Yeh, 2004), as well as be assessed on them. To assess the students’ self-assessment skills is 

of central importance since the assessment has a very strong influence on students’ learning (Brown et al., 1997). 

Examination schemes in profession-directed education traditionally provide educators with a thorough insight into 

students’ profession-related skills and competences, but little is known about students’ ability to self-assess their 

proficiency, to define their own learning objectives, and independently direct their competence development during 

their professional life. A structured assessment methodology focused on such metacognitive skills at the side of 

traditionally examined skills and knowledge, would therefore be a very important tool in higher education. 

De la Harpe and Radloff (2000) give several examples of both qualitative, like learning logs and interviews, as well 

as quantitative methods, mainly Likert scale based, to assess metacognitive skills. Most of these methods are, 

however, not integrated in the learning activities in an authentic manner, and the authors also point to the fact that 

“Students may be reluctant to engage in activities that focus on learning rather than on course content and may not 

devote the time and effort needed to complete assessment tasks effectively” (p. 177). 

To avoid this pitfall, the “Interactive Examination”, a structured assessment methodology developed and evaluated in 

the Faculty of Odontology at Malmö University, Sweden, has included the assessment of self-assessment skills in a 

regular examination. The methodology aims to evaluate students’ content specific skills and competences in parallel 

to their self-assessment skills, while expanding and supplementing the learning process. The self-assessment skills 

are assessed with both quantitative as well as qualitative means. Also, the methodology makes use of modern 






17

information- and communication technology in order to facilitate training and feedback without necessarily 

increasing the workload of the personnel (Mattheos et al., 2004b). 

The present study aims to describe the model of the Interactive Examination and present the results from a 

multicentre evaluation study with undergraduate students in the Faculty of Odontology (OD) and School of Teacher 

Education (LUT) at Malmö University. It should be emphasized from the start, however, that this study does not aim 

for direct comparison of the two student groups, as differences in educational context and experimental settings 

would make this task meaningless. Rather, what is attempted is a “parallel execution”, where differences and 

similarities in the two institutions can be identified, leading to improvements of the methodology, as well as giving 

rise to new questions for further investigation. 

Material and method 

General Principle of the “Interactive Examination” 

In principle, the methodology is based on six explicit stages: 

1. Quantitative self-assessment. At the beginning of the process, the students assess their own competence through a 

number of Likert-scale questions, graded from 1 (poor) to 6 (excellent). In addition there are three open text fields, 

where the students can elaborate further on their self-assessment. When possible, the self-assessments are compared 

with the instructors’ judgements of students’ competence, and feedback is given – a process that to some extent can 

be automatized by the software. The purpose of this comparison is to highlight differences between student’s and 

instructor’s judgement, and not to constitute a judgement per se. Possible deviations between self-assessment and 

instructor’s assessment are only communicated to the students as a subject for reflection or a possible discussion 

issue with the instructor. 

2. Personal task. After the completion of the initial self-assessment, students receive a personal task in the form of a 

problem which they might experience during their professional life. This is an interactive part of the examination, 

where the interaction takes place between the student and the different affordances provided (such as links, pictures, 

background data etc.). The students have to come up with a solution strategy and elaborate their choices in written 

text. 

3. Comparison task. After the personal task, the students receive a document representing the way an “expert” in the 

field chose to deal with the same task. This “expert” answer does not correspond to the best or the only solution, but 

rather to a justified rationale from an experienced colleague, which remains open to discussion. The “expert” 

documents have been written in advance and the students are given access to them as they submit their responses to 

the personal task. This is a way of dealing with the problem of providing timely feedback to a large number of 

students, but the “expert” answers also provide a kind of social interaction, although in a fixed (or “frozen”) form. 

The stance taken here is thus that, although interaction is needed in order for learning to take place, this interaction 

does not necessarily involve direct communication or collaboration between humans (cf. Wiberg in this issue), but 

the interaction could also be mediated by technology. 

By the aid of the “expert” answer, the students can, according to the concept of “the zone of proximal development” 

(Vygotsky, 1978), potentially reach further than they can on their own, thus making the assessment dynamic. 

Dynamic assessment means that interaction can take place, and feedback can be given, during the assessment or 

examination, which separates it from more ”traditional assessments” (Swanson & Lussier, 2001). In this way, 

dynamic assessment provides the possibility to learn from the assessment, but also to assess the student’s potential 

(”best performance”), rather than (or together with) his or her ”typical performance” (Gipps, 2001). Empirical 

studies has shown that dynamic assessment indeed help to improve student performance, and also that lowperforming 

students are those who benefit the most, thus making the difference between high- and low-performing 

students less pronounced (Swanson & Lussier, 2001). 

After receiving the “expert” document, the students must, within a week, prepare a comparison document, where 

they identify differences between their own and the “expert” answer. The students are also expected to reflect on the 

reasons for these differences and try to identify own needs for further learning. This comparison document is a part 

18

of the qualitative self-assessment in the Interactive Examination, which, in contrast to the quantitative selfassessment, 

is used for summative purposes as well. 

4. Evaluation. After the examination the students evaluate the whole experience through a standardized form. At this 

point students have no feedback whether they have successfully completed the exam or not. 

5. Assessment of students. The students are assessed on the basis of: (1) their competence and knowledge on course 

specific objectives, and their ability to relate theoretical knowledge to displayed scenarios and critical thinking, as 

expressed in their personal task, as well as (2) their ability to reflect on their choices, identify weaknesses and define 

future learning objectives, as expressed in the comparison document. 

When poor performance is demonstrated in any of the above fields, students are assigned additional tasks. In this 

way students cannot “fail” the exam completely, but might be requested to practice and improve the respective skills, 

until a satisfactory level of competence is reached. 

6. Personalized feedback. One month after the examination, individual feedback is sent electronically to all students. 

This feedback includes comments on students’ self-assessment and how it relates to the judgement of the clinical 

instructor, as well as comments on the personal task and the comparison document. Finally the feedback contains 

suggestions for future tasks if necessary. 

Current Experimental Settings in OD and LUT 

The current experimental settings, as presented below, show how the Interactive Examination was applied in the 

autumn 2004 to undergraduate students at OD and LUT, both which are faculties at Malmö University. OD was 

founded in 1946 and provides undergraduate education in Dentistry, Dental Technology, and Dental Hygiene. The 

Interactive Examination is used within the dentistry programme, where 40 students are accepted every autumn. 

Within this programme, Problem-Based Learning (PBL) has been used since 1990 (Malmö University, 2006). LUT, 

with approximately 8.000 students, is the largest faculty at the university, and the undergraduate education covers the 

whole range from pre-school teaching to the upper secondary level. The undergraduate education at LUT is 

organized in five major areas, or fields of knowledge, and the Interactive Examination is used within the field called 

Science, Environment and Society (Malmö University, 2007). The Interactive Examination was made available over 

the Internet through e-learning platforms, making it possible for the students to do the examination at any place they 

found suitable. 

The platforms used are non-commercial and both have been developed locally to meet the requirements of the 

specific learning activities to take place, as well as to facilitate the research conducted. At LUT a platform called 

ALHE (Accessability and Learning in Higher Education) has been used, and this will be the one presented more 

thoroughly in this article. ALHE is in many respects a conventional educational platform with both asynchronous 

(discussion forums and e-mail) and synchronous (online chat) communication tools, but it also includes some quite 

specific features. For example, students’ discussions are logged and displayed in a way to help the students reflect 

upon their own dialogic pattern and mutual knowledge building (e.g. who communicates with whom and to what 

extent, what kind of contributions have the students made), but also to facilitate research on the same issues. Another 

feature is the use of questionnaires, where the results can be exported to data sheets (such as Microsoft Excel or 

SPSS) for further analysis. Furthermore, ALHE is built to allow for the addition of new modules, and the Interactive 

Examination is such a module that has been implemented into the main platform. In the teacher interface, the files 

necessary for the examination (e.g. movies and “expert” documents) can be uploaded in a sequence similar to the 

methodology, and the files are then accessible in the student interface as hyperlinks. This is then complemented with 

the use of questionnaires in the quantitative self-assessment and the student evaluation as described below. 

1. Quantitative self-assessment. Both students of OD and LUT started the Interactive Examination through a 

questionnaire-based self-assessment in specific course directed competencies. 

Each OD student was assigned to one of six clinical instructors. The clinical instructors held regular meetings, where 

they revised the learning objectives for the 3rd semester and the quality assessment criteria for clinical work. As a 

result, 11 specific self-assessment fields were prepared for the students, reflecting 11 basic competencies. In the 

19

Interactive Examination, the students self-assessed their competence in the 11 knowledge and skill areas through an 

on-line form connected to a database. The self-assessment was carried out through ordinal scales marked from one to 

six, with six marked as “excellent” and one being “poor”. The same form had been used by the clinical instructors 

when assessing students’ clinical competence at the end of the semester. The results from students’ self-assessment 

were later compared to those originating from their clinical instructors. 

The self-assessment form LUT students completed was based on a scoring guide, or rubric, which was developed for 

this particular examination. The criteria in the rubric were equivalents to the self-assessment questions, making 

possible a comparison of students’ self-assessment to their actual results. The questionnaire had 13 questions relating 

to basic teacher competencies and three questions regarding reflection and self-assessment skills. 

2. Personal task. The OD students received a clinical patient case, accompanied with the possibility to access 

relevant images and diagnostic data. Their task was to identify the problem (diagnosis) and propose a treatment plan. 

Figure 1 illustrates the personal task, on the left side links to movies and other affordances are visible just below the 

Swedish title “Interaktiv examination”. The three forms described in the text are seen in the central portion of the 

screen. The right hand picture shows how one of the movies is displayed in the browser by using Flash software. 

Movies were also available as mpeg-files, with higher resolution and higher-quality sound, for those students not 

using a dial-up internet connection. 

Figure 1. Screenshots from the student interface in the ALHE version of the Interactive Examination 

The LUT students watched short movie sequences showing different problematic situations in a classroom context. 

Along with the movie sequences, the students could access some background data for the situation displayed, as well 

as the dialogue in text format. Of the total examination time, about one hour per movie was allocated for this part. 

With the movies as a starting point, the students filled in three different forms on the screen (see Figure 1): 

i. Describe the situation objectively and without prejudice, 

ii. Analyze the displayed situation on the basis of relevant literature and knowledge developed in the course, and 

iii. Consider different alternatives and give a proposal of how the teacher in the movie sequence should act. 

3. Expert response and comparison document. Both OD and LUT students received a text representing the way a 

qualified colleague chose to deal with the same problem. All students had to come up with a written reflection as 

directed by the previously described principles. 

4. Evaluation. After the examination the students evaluated the whole experience through a standardized form. The 

form included ten fields to which students could respond to on an ordinal scale from 1-9, as well as some multiple 

choice questions and free text fields. Free text comments were possible at the side of all fields. Eight fields were 

identical in the two centres, and two fields were similar. Along with practical issues, the form contained questions 

about the examination as a learning experience from the students’ point of view and the perceived relevance for their 

future profession. 

20

5. Assessment of the students. The students were assessed on an “acceptable”/”not acceptable” basis, depending on 

their performance in the written task and the comparison document. In the Faculty of Odontology one assessor 

evaluated all personal tasks and comparison documents, whereas six clinical instructors (4 female - 2 male) provided 

judgements for comparison with students’ initial self-assessment. The personal task was assessed through specific 

discipline related evaluation criteria, while the comparison document was assessed through a specific scoring guide 

(Table 1). 

At LUT, a scoring rubric covering the personal task as well as the comparison document was developed to provide 

information to both assessor and students what was to be assessed (Table 2). The students had access to the rubric 

well before the examination to enable a discussion of the assessment criteria with their instructors. The guide was 

also thought to make possible a more reliable assessment, despite the complexity of the task. All examinations were 

assessed by an external assessor. 

6. Personalized feedback. One month after the examination, individual feedback was sent to all students. Feedback to 

the OD students included their performance in the written task, commentary on their comparison document, as well 

as suggestions for future learning. 

For the LUT students, examination results were provided for each criterion in the scoring rubric. This very specific 

feedback showed both which criteria to give more attention in the future, as well as the direction of that attention to 

steer future learning. The students, as well as the researcher, could also easily compare the initial self-assessment to 

the actual results. 

Sample 

Both studies were carried out in the autumn 2004, with a cohort of first year student teachers in science and 

mathematics and second year dental students respectively. While all dental students were included in the study 

(n=34, 18 female- 16 male), some student teachers did not show up for the exam (n=171 out of 174, 103 female- 68 

male). Also some student answers are missing in the first part of the examination, due to technical problems, making 

the LUT sample somewhat smaller for the self-assessment (n=166). All students in both centres were exposed to the 

Interactive Examination for the first time. 

Statistical Analysis 

Students’ responses on the self-assessment fields were compared for agreement to those from their clinical 

instructors (OD), or to the actual examination results (LUT), using a two tailed Wilcoxon’s signed rank test. This test 

is a non-parametric analogue to the t-test and is used to determine whether two paired sets of data differ significantly 

from each other. The comparison was carried out for each individual student and also independently for each of the 

self-assessment fields. A frequency analysis was performed for the total of students’ and instructors’ scores in the 

assessment forms. The potential influence of gender or instructor on the examination scores, as well as the pattern of 

the self-assessment (higher, lower or in agreement), was investigated with regression analysis. Non-parametric linear 

regression was used to correlate gender, group and examination scores with students’ pattern of self-assessment. 

Qualitative Analysis 

The qualitative analysis of the students’ answers to the comparison task aims to answer the following questions: 

1. What kinds of differences or similarities between student and “expert” answers were identified? 

2. Which reasons, for the identified differences, are stated? 

3. Which weaknesses in their own competence as a teacher or dentist, and which learning needs, are identified 

by the students? 

21

Table 1. Criteria for grading OD students’ comparison documents 

Evaluation Excellent (3 pts) Acceptable (2 pts) Not acceptable (1 pt) 

Comparison of content The student has 

identified most/all the 

Analysis explanation 

of the differences 

Defining learning 

objectives 

important differences 

The student is able to 

analyze/attribute 

differences 

The student reaches the 

learning objectives 

deriving from the 

analysis of differences 

The student has identified 

half of the major differences 

The student can only partly 

analyze/attribute differences 

The student provides learning 

objectives only partly 

relevant to his analysis of 

differences 

The student has only 

identified very few or 

irrelevant differences 

The student does not 

attempt to analyze the 

differences. 

The student does not reach 

learning objectives, or they 

are irrelevant to his 

analysis of differences 

Evaluation 

Table 2. Part of the LUT scoring guide 

Acceptable Excellent 

Reflection: 

The reflection identifies differences The reflection identifies most of, or all, 

Can you use your own, between the own and the other teacher’s relevant differences between the own 

as well as others’, interpretation of the situation. and the other teacher’s interpretation of 

experiences as a basis 

for reflection and 

development? 

the situation. 

The reflection presents some reason, or 

reasons, for the identified differences. 

The reflection identifies shortcomings in 

own professional competence. 

The reflection argues in favour of own 

standpoints on the basis of relevant 

literature. 

The reflection identifies shortcomings in 

own professional competence and states 

learning needs resulting from these 

shortcomings. 

In a previous study (Mattheos et al., 2004b) the differences which students identified in the comparison of their own 

answers to the “expert”, were categorized into differences in (1) form, (2) content, and (3) attitude towards the 

content. This classification is used in the current study as well, with the addition of a fourth category which was 

mainly present among LUT students: Differences in interpretation. Difference in interpretation occurs when the 

student and the “expert” has interpreted the same situation in totally different ways, so that attempts to compare the 

two are extremely difficult. One example is when a student interprets the situation as a gender issue, while the 

“expert” writes about the same situation from an assessment point of view. Due to the nature of the problem dealt 

with in the cases of the OD students, differences in interpretation were much less prevalent. 

Results 

Evaluation – Students’ Attitudes 

Students’ acceptance of the methodology was positive, with the median values in most evaluation fields lying 

between 6 and 8 (OD), or around 6 (LUT), on a scale to 9 (Table 3). Based on their free text comments, OD students 

favoured the opportunity to reflect on one’s own self-assessment and the contact with their educators in this type of 

assessment. Some students would prefer more timely and personal feedback. 

Student teachers appear to have found this mode of examination interesting and instructive, especially the 

comparison part. Some mentioned enhanced motivation and engagement as a consequence of the authentic tasks. A 

few students wrote about less stress and anxiety as compared to more traditional modes of examination. Negative 

comments were mainly about lack of background information for the movie situations. 

22

Table 3. Some results from the student evaluation 

Question Student response OD 

(median) 

How do you value the Interactive Examination as a 

8 

learning experience? 

1 (not effective) - 9 (very effective) 

(n = 33) 

Was it clear what was expected from you in the 

6 

Interactive Examination? 

1 (very unclear) - 9 (very clear) 

(n = 33) 

To what extend do you feel you got the chance to 

7 

show what you know? 

1 (very little) - 9 (very much) 

(n = 33) 

How much do you think this type of examination can 

6 

help you to prepare for your working tasks as a 

dentist/teacher? 

1 (very little) - 9 (very much) 

(n = 33) 

How difficult were the examination cases? 

6 

1 (very easy) - 9 (very difficult) 

(n = 33) 

Self-Assessment 

Student response LUT 

(median) 

6 

(n = 140) 

5 

(n = 138) 

6 

(n = 140) 

6 

(n = 139) 

6 

(n = 138) 

The students’ self-assessment forms provided a total of 369 scores from 34 students (OD) and 2647 scores from 166 

students (LUT). The responding forms from the clinical instructors at the OD amounted to 374 scores, as some 

students had probably omitted some of the evaluation fields. The unmatched scores were excluded and the 

comparison was based on 369 scores from 34 students. 

A total of 142 (38 %) dental student scores were higher than the judgement of the clinical instructors, while 88 (24 

%) were lower and 139 (38 %) were in agreement. On an individual basis, 12 students’ (35 %) judgement was 

significantly higher than that from their instructors (p

content. Differences in the attitude were rarely reflected in the defined future learning objectives. On the other hand, 

the attitude differences were the field where students would most likely choose to defend their choice and argue 

against the response of the “expert”. Differences in the interpretation were something that was very rarely 

encountered by the OD students. 

In the majority of cases the OD students proved to be very skilful in locating the weak points and gaps in their 

knowledge, as opposed to the LUT students. Six students (2 females - 4 males) failed to identify the actual problems 

with their essays and were assigned some additional tasks. 

Discussion 

The main focus of this study was the reflective process which is initiated through comparing your own work with 

that of someone else. This process is well rooted in students’ self-assessment ability, a necessary professional skill. 

The Interactive Examination is a methodology developed on this principle and has been carried out with several 

cohorts of students with promising results (Mattheos et al., 2004a; Mattheos et al., 2004b). 

The present application of the Interactive Examination is unique in the sense that it brings together two different 

educational environments. As was emphasized initially, this study does not aim for direct comparison of the two 

student groups, but rather for a “parallel execution”, investigating the self-assessment pattern and the acceptance by 

the students in two different institutions. Furthermore, the study aimed to identify institutional differences and 

similarities, hopefully leading to improvements of the effectiveness and applicability of the methodology, as well as 

providing new insights into the respective institutional learning cultures. 

The students appeared to receive this form of examination favourably in both institutions. Students’ appreciation and 

acceptance of the examination methodology, as well as the value of the reflective process, are seen as prerequisites in 

order to affect their learning. The positive experience of the students therefore justifies a further analysis and 

discussion of the results from the Interactive Examination. 

In the studies reported here, there was a somewhat different pattern of self-assessment in the two centres. While a 

large portion of the scores at OD were in agreement with the judgements of the clinical instructors, the corresponding 

value at LUT was much lower. Also, whereas the dental students had a number of self-assessment scores both higher 

and lower than the instructors; the by far greatest number of scores from the student teachers were higher than the 

examination results. It should be kept in mind, however, that the self-assessment was somewhat different in the two 

centres. While the OD student scores were compared to their instructors’ judgement, the LUT students’ scores were 

compared to their examination results. This difference, along with other contextual factors (such as the formulation 

of the self-assessment questions), have a potential influence on the students’ self-assessment pattern. Also, as OD 

students and instructors have been spending a whole semester together, a “calibration” effect might bring their 

judgements closer to each other’s. This means that the differences between student teachers and dental students must 

be interpreted with caution. There are, however, striking differences in the frequency of scores in agreement, as well 

as in the distribution of higher and lower scores, warranting further discussion and research. 

According to previous research on self-assessment, low-ability students many times overestimate their grade or score 

as compared to the teachers’ judgement, while high-ability students more often are in agreement with their teacher 

(Kruger & Dunning, 1999). Also, progress in the course of studies seems to affect the self-assessment skills, where 

students in the beginning of a course produce less reliable assessments of themselves (Topping, 2003). In OD there 

was no relation between the students’ success on the exam and their self-assessment pattern, while on LUT, there 

was a relation between self-assessment pattern and success on the exam. A possible explanation to this, is that the 

dental students both have progressed somewhat further in their education (they were on the 3rd semester of their 

studies), but might also be more homogenous and calibrated as a group. By the 3rd semester the dental students have 

already spent a considerable amount of time working together in an environment where peer learning and group 

dynamics have a significant role, as well as continuous contact with the instructors. 

No other factor, such as students’ gender, group or clinical instructor, could be found relating to the self-assessment 

pattern in either OD or LUT. Studies reported in the literature on self-assessment shows no uniform results on gender 

differences in self-assessment skills either (Arnold et al., 1985; Ericson et al., 1997; Topping, 2003). 

24

The qualitative analysis of students’ comparison documents provided some very interesting findings. Evidently, the 

dental students were primarily focused on differences with the “solution” provided, while student teachers seem to 

have extensively focused on similarities. Similarities tend to be only briefly mentioned, if at all, by dental students 

and are usually not accompanied by further arguments. It appears that the dental students treat the similarities as 

something well expected, almost self-evident, not worth of special attention and choose to focus on the explanation 

of differences instead. This attitude has been repeated almost in every cohort of dental students so far, to the extent 

that the assessors in the dental faculty have considered this a standard attitude, the reasons of which were never 

questioned. However, the execution of the Interactive Examination in the Teacher Education has brought a valuable 

insight in this field. In addition, the differences in interpretation are encountered in student teachers’ documents, 

while they are rarely observed with dental students. Besides contextual factors, such as the nature of the expert 

document, one might consider many reasons as likely to have contributed to these differences: 

Different nature of the assessed task. Diagnosis and treatment planning as encountered in the second year dental 

students’ cases require a well defined array of knowledge fields and competences. Although controversies are very 

often encountered, the existence of specific guidelines and accepted practices, limits down the spectrum of viable 

choices as well as the importance of subjective factors. Dental students in their great majority identified the same 

main problem in each clinical case. With most students having the same starting point, it might be that differences 

are more likely to attract attention than similarities. 

On the other hand, the task which student teachers were called to complete covered a wider area of subjects, 

including social and moral issues, where application of standards and guidelines is sometimes unclear. Furthermore, 

the cases could be approached from different points of view, defining different problems as starting points. This 

resulted in different intervention strategies and differences in interpretation. 

Difference in the institutional learning cultures. It appeared that dental students tend to see more authority in the 

“qualified dentist” than student teachers see in a “qualified teacher”. Dental students, at least at this early stage of 

their studies, seem to be less eager to question the opinions of the qualified dentist than student teachers of an 

experienced teacher. If this is true, it might reflect differences in how the students see their future role as “end 

products” of their education. 

A dentist might represent for the dental students a very strictly defined set of competences, accompanied by a certain 

degree of “authority”, which they are most likely uncomfortable to challenge. Student teachers, however, seem to 

adopt more of a peer attitude towards their qualified colleagues. This is reflected in the fact that some students has 

chosen to criticize the “expert”. The criticism is mainly about views and values, but to some degree also about the 

interpretation and the examples chosen. In other cases the students regard their own solutions as being qualitatively 

better than the qualified teacher’s. Here it is foremost choices of specific examples or actions taken that are 

considered better. 

In future studies it would be very interesting to further investigate this assumption and see if there are certain 

differences between students of different profession-directed educations, in terms of how students perceive their 

development towards the “final product” of their studies. 

Conclusions 

The added value of this multicentre study is threefold. The first lies in the validation of the methodology. There is 

often a problem in estimating the quality of new modes of assessment, since they cannot always be evaluated on the 

basis of traditional psychometric criteria. Gielen et al. (2003) argue that “To do right to the basic assumptions of 

these assessment forms [“authentic assessment” and “performance assessment”], the traditionally used psychometric 

criteria need to be expanded, and additional relevant criteria for evaluating the quality of assessment need to be 

developed” (p. 38). In this widened set of criteria, referred to as “edumetric” criteria, the validity concept has been 

expanded to include the tasks used, considering authenticity and complexity in relation to the knowledge domain 

being assessed, but also consequences of the assessment such as the influence on students’ learning or learning 

strategies (Sambell et al., 1997; Gielen et al., 2003). 

25

Even though further research on the quality of the Interactive Examination is needed to better determine the 

consequences of the methodology, in terms of students’ learning and learning strategies, efforts have been made to 

include features aiming specifically for self-assessment skills and thus trying to make the examination more valid for 

the proposed purpose (Frederiksen & Collins, 1989). As described earlier, two parts of the Interactive Examination 

involves self-assessment. First, the students estimate their own competence according to Likert-like questions, where 

the results are compared either to judgements from the instructor (OD) or the actual examination results (LUT). This 

comparison, however, does not constitute a judgement per se and possible deviations between self-assessment and 

results are used only to draw the students’ attention to the difference and thus make reflection and learning possible. 

Secondly, the students compare their answers with the answer of an “expert”. This comparison is assessed, and 

feedback is given, according to scoring criteria. 

But addressing self-assessment skills is not the same thing as really capturing them. However, by investigating how 

the methodology is used and perceived in the two different institutions, an estimation of the validity can be made. 

Even though there are some institutional differences, displayed for instance in the different ways to handle the 

comparison task by LUT and OD students, the overall applicability of the methodology is similar in both centres and 

the students respond to it in a analogous manner. This indicates that the Interactive Examination might be a valid 

methodology for assessing students’ self-assessment skills in authentic settings, and thus a potential tool for assisting 

the development of certain metacognitive skills in higher education. 

The second added value of cross-sectional, multicentre studies such as this, is the provision of a better insight to 

students’ self-assessment abilities. Future studies could investigate the longitudinal changes of students’ selfassessment 

abilities throughout the curriculum. Such follow-up studies are necessary in order understand how these 

skills evolve and also to allow educators to design proper interventions in order to early identify and support students 

with weak self-assessment abilities. 

The last added value to be commented upon, relates to the use ICT in the Interactive Examination. Information and 

communication technology is used in several ways in the examination methodology, and for several reasons. For 

example, it makes possible a automatized comparison of the quantitative self-assessment and instructors’ judgement, 

and it also provides the necessary interactivity in the personal task, where the dental students have access to relevant 

images and diagnostic data, and the student teachers watch movie sequences that has to be accompanied with links to 

background data and other affordances. In both cases the students need to access the “expert” document after 

submitting their personal task, while at the same time saving their answers to the personal task in a database 

available to both assessors and researchers. Most importantly, however, the use of technology makes it possible to 

make valid assessments of student competences in a way not possible without this technological support. For 

instance, the authenticity of the examination could not be brought about by a paper-and-pencil test (cf. Lam, 

Williams, & Chua, 2007), nor could the same effectiveness be achieved if the students were assessed while actually 

performing in practice – this is especially true for the teacher education with such a large number of students. A 

conclusion is thus that training and valid assessment of self-assessment skills can be facilitated through the 

Interactive Examination, and that this can be done without necessarily increasing staff numbers or workload. In 

addition, as the examination is available online, the methodology could easily be used for distance education 

purposes. The Internet accessibility was used by a majority of the LUT students who preferred to carry out the 

examination at home or, in a few cases, from other parts of the world (e.g. Afghanistan and Iceland). 

References 

Arnold, L., Willoughby, T., & Calkins, E. (1985). Self-evaluation in undergraduate medical education: a longitudinal 

perspective. Journal of Medical Education, 60 (1), 21-28. 

Brown, G., Bull, J., & Pendlebury, M. (1997). Assessing student learning in higher education, London: Routledge. 

De la Harpe, B., & Radloff, A. (2000). Informed teachers and learners: The importance of assessing the 

characteristics needed for lifelong learning. Studies in Continuing Education, 22 (2), 169-182. 

Ericson, D., Christersson, C., Manogue, M., & Rohlin, M. (1997). Clinical guidelines and self-assessment in dental 

education. European Journal of Dental Education, 1 (3), 123-128. 

26

Fredriksen, J. R., & Collins, A. (1989). A systems approach to educational testing. Educational Researcher, 18 (9), 

27-32. 

Gielen, S., Dochy, F., & Dierick, S. (2003). Evaluating the consequential validity of new modes of assessment: The 

influence of assessment on learning, including pre-, post-, and true assessment effects. In Segers M., Dochy F., & 

Cascallar E. (Eds.), Optimizing new modes of assessment: In search of qualities and standards, Dordrecht: Kluwer 

Academic Publishers, 37-54. 

Gipps, C. (2001). Sociocultural aspects of assessment. In Svingby, G. & Svingby, S. (Eds.), Bedömning av kunskap 

och kompetens, Stockholm: Lärarhögskolan i Stockholm, PRIM-gruppen, 15-67. 

Hays, R. B., Jolly, B. C., Caldon, L. J., McCrorie, P., McAvoy, P. A., McManus, I. C., & Rethans, J-J. (2002). Is 

insight important? Measuring capacity to change performance. Medical Education, 36 (10), 965-971. 

Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one's own 

incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77 (6), 1121-1134. 

Lam, W., Williams, J. B., & Chua, A. Y. K. (2007). E-xams: harnessing the power of ICTs to enhance authenticity. 


Malmö University (2006). About the Faculty, Retrieved October 1, 2007, from, http://www.mah.se/templates/ 

Page____13096.aspx. 

Malmö University (2007). About the School, Retrieved October 1, 2007, from http://www.mah.se/templates/ 

Page____13082.aspx. 

Mattheos, N., Nattestad, A., Christersson, C., Jansson, H., & Attström, R. (2004a). The effects of an interactive 

software application on the self-assessment ability of dental students. European Journal of Dental Education, 8 (3), 

97-104. 

Mattheos, N., Nattestad, A., Falk Nilsson, E., & Attström, R. (2004b). The Interactive Examination: Assessing 

students’ self-assessment ability. Medical Education, 38 (4), 378-389. 

Ngan, P., & Amini, H. (1998). Self-confidence of general dentists in diagnosing malocclusion and referring patients 

to orthodontists. Journal of Clinical Orthodontology, 32 (4), 241-245. 

Reisine, S. (1996). An overview of self-reported outcome assessment in dental research. Journal of Dental 

Education, 60 (6), 488-93. 

Sambell, K., McDowell, L., & Brown, S. (1997). "But is it fair?" An exploratory study of student perceptions of the 

consequential validity of assessment. Studies in Educational Evaluation, 23 (4), 349-371. 

Swanson, H. L. & Lussier, C. M. (2001). A selective synthesis of the experimental literature on dynamic assessment. 

Review of Educational Research, 71 (2), 321–363. 

Topping, K. (2003). Self and peer assessment in school and university: Reliability, validity and utility. In Segers M., 

Dochy F., & Cascallar E. (Eds.), Optimizing new modes of assessment: In search of qualities and standards, 

Dordrecht: Kluwer Academic Publishers, 55-87. 

Vygotsky, L. S. (1978). Mind in society, Cambridge: Harvard University Press. 

Yeh, Y-C. (2004). Nurturing reflective teaching during critical-thinking instruction in a computer simulation 

program. Computers & Education, 42 (2), 181-194. 

27

Olofsson, A. D. (2007). Participation in an Educational Online Learning Community. Educational Technology & Society, 10 (4), 

28-38. 

Participation in an Educational Online Learning Community 

Anders D. Olofsson 

Umeå University, Sweden // Tel: +46 90 786 78 09 // Fax: +46 90 786 66 93 // Anders.D.Olofsson@pedag.umu.se 

ABSTRACT 

This paper discusses the issue of learner participation in a net-based higher education course. With the starting 

point in recent educational policies formulated by the European Union and the results of an evaluation report 

from the Swedish Net University, I raise the question of which pedagogical aspects need to be considered in 

order to support active learner participation in these types of learning environments. Based on analyzed data 

from 19 semi-structured interviews with trainees on a Swedish net-based teacher training programme supported 

by Information and Communication Technologies, I attempt to show that in order to become a member of such 

educational online learning community, each trainee is required to be active and hold an inclusive attitude 

towards the other members. Further, it seems that the trainees often had to rely on and trust each other due to the 

sparse communication with their teacher trainers. I conclude this paper by discussing the need for a pedagogical 

approach that relies heavily on social, collaborative and ethical aspects of learning as a starting point for the 

design of online learning communities to support the kind of education needed for the 21st century. 

Keywords 

Distance Education, Learning Community, Participation, Pedagogy, Teacher Training 

Introduction 

In Lisbon, the European council stated that the European Union (EU) should become the world’s most competitive 

and dynamic knowledge-based economy (EU legislation, 2003a). The field of education has been identified as one of 

the target areas for the implementation of this declaration, thus demanding an elaborate educational strategy. In 

response to this demand, a number of tangible goals for how the educational system in Europe should be carried out 

in the future (EU legislation, 2003b) have been identified. These goals have been defined as follows; first, to improve 

the quality in the education system, second, to facilitate the access to the education system for members of the 

European Union and third, to open up the educational system in the European Union to the rest of the world. 

Sweden has been actively working in this field focussing on the creation of an extended number of university 

programmes and courses following the EU recommendations. Programmes and courses which are provided offcampus, 

using the internet as means for participation, collaboration and dialogue have increased significantly during 

the last decade (Lindberg & Olofsson, 2006). 

Recently, The Swedish National Agency for Higher Education (2005) stated in an evaluation of the Swedish Net 

University, that net-based education could be understood as an educational form that contributes to widen the base 

for recruitment to higher education. Higher education in Sweden has traditionally had problems recruiting students 

that do not have middle and upper class backgrounds. Additionally, the evaluation stated that the students’ 

geographical location was less important within this educational form. This is claimed to be due to the possibility to 

attend university programmes and courses all over Sweden, using the internet. It seems for that reason no longer 

necessary for the students to physically attend the university campus a number of times each semester. Higher 

education today appears to be open for people from every walk of life in Swedish society. This trend within higher 

education is however not revolutionary and has for example, according to Berg (2003), been present in the USA for 

quite a long time. 

Nevertheless, one issue that probably needs to be noted as an area for re-thinking is how to organize such higher netbased 

education. How is it possible to convert the net-based education into a venture characterized by active 

participation, collaborative learning, negotiating and sharing of meaning (Wenger, 1998)? What kind of pedagogy 

could be used to support and sustain the knowledge-building process of each student, and at the same time guarantee 

that education does not become merely a question of transferring information? How can social aspects of learning be 

included in the design of net-based education and what are some of its pitfalls? This paper intends to investigate 

some possibilities within a net-based teacher training programme in which the students are organized in smaller 

study-groups, which here are understood as Educational Online Learning Communities (E-OLCs) (Carlén & Jobring, 

2005; Olofsson & Lindberg, 2006). I wish to especially shed light on issue of values underpinning all activities 






28

within the Educational OLCs. The overall research question is formulated as; what is required from each teacher 

trainee to become a member of one of the Educational OLCs within the programme? 

Additionally, I will describe in further detail how the understanding of distance education has moved from being a 

question about transferring knowledge, towards being a question about learning together in an E-OLC. Thereafter I 

describe the results of a study conducted with 19 trainees on a Swedish net-based teacher training programme 

supported by Information and Communication Technologies. I will conclude by discussing the outcomes of the study 

in connection to ideas discussed in section two and the challenges that might be present in the design of Online 

Learning Community (OLC). The paper concludes with stressing the importance of including pedagogical issues, 

firmly based in an ethical view of people and education, when educating online. 

From Individuality to Community – Net-based Learning in Transition 

Traditionally, distance education has been an educational form with limited possibilities for the students to learn with 

and from each other (Howard, Schenk & Discenza, 2004). This means that the students often were reduced to using 

technology that only, more or less, made it possible for them to download ready-made educational material within 

the programme or course they attended (Bonk & Cunningham, 1998). If the students wanted to collaborate with their 

co-students, for example, in order to gain a deeper understanding or to collaborate they had to use ordinary letters, 

telephone or even to psychically attend the university campus. 

Today this picture has changed. A new movement within higher education is present, which is also expressed in 

legislative documents regarding education in Europe. For example, the recommendations “eLearning – Designing 

tomorrow’s education” (EU legislation, 2003a) states the need for establishing virtual forums and campuses that will 

promote the development of distance teaching and learning and the exchange of best practice and experience. This 

call for virtual forums can be seen as a result of research investigating the concept of community within education, 

teaching and learning. 

Community, however, has been a concept used for different purposes within research for a long time, but the 

renewed interest stems from the work on communities of practice initially carried out by Lave & Wenger (1991) and 

further developed by Wenger (1998). Learning is to be understood as the movement or trajectory, in which a process 

of legitimate peripheral participation is developed between people in a certain context. People, or members, in a 

community are being fostered into the core of the practice by processes characterized by negotiating of meaning. 

Processes that not only include aspects of how something should be learnt and understood but also aspects of 

inculcation of values, assumptions and beliefs that are underlying the practice (see also Sergiovanni, 1999) ensuring 

a common ground for the activities within the community (Bauman, 2001). 

The concept of community has lately been used in relation to aspects on education and learning located on the 

internet (Sorensen & Ó Murchú, 2004) which is referred to as OLC (Jaldemark, Lindberg & Olofsson, 2005, 

Lindberg & Olofsson, in press; Lock, 2002; Olofsson & Lindberg, 2005; Palloff & Pratt, 2005; Seufert, Lechner & 

Stanoevska, 2002). An OLC is said to cross geographical borders and the aspect of time as a barrier is reduced 

(Haythornthwaite, Kazmer, Robins & Shoemaker, 2000; Lewis & Allan, 2005). According to Carlén and Jobring 

(2005), there are different types of OLCs depending on their focus. The first type is the so-called Professional OLC, 

which focuses on work-related questions. The second type is the so-called Interest OLC, which focuses on different 

topics related to the members’ leisure activities. The third type is the so-called Educational OLC (E-OLC) which 

focuses on different educational issues. 

The concept of community seems to be very promising in relation to higher net-based education, not only today but 

also tomorrow. However, when considering the concepts of community and OLC, the impression is one of a rather 

un-complicated process of both becoming and being a member of a community, regardless if the community is 

located on the internet or not (compare Söderström, Hamilton, Dahlgren & Hult, 2006). The rationales behind the use 

of community seems to be that it is only necessary for people to connect to each other (perhaps through the internet) 

for a process of bonding, building a community, and a process of collaborative learning by means of active 

participation and dialogue to appear. Is it that by interconnecting people in a forced or fictive community, in this 

case, teacher trainees in an E-OLC, a joint learning enterprise is apparent and learning occurs? What aspects of the 

moral underpinnings and the ethical considerations underlie a membership within such an E-OLC? 

29

Etzioni (1993) points out that an important aspect of a community is shared morality. The community speaks to its 

members with a moral voice, restricts and sets demands on its members. Within a community there are always, 

according to Etzioni, possibilities for negotiations between members; negotiations that may decide which activities, 

behaviours, opinions, values etc that should be considered right or wrong within the community. If any violation or 

obstruction in relation to the negotiated meaning of being a member occurs, the community will respond with power 

in order to stop the un-wanted, non-sanctioned activities. 

Bauman (2001) describes in a similar way the double-edged character of the community; how it cuts both ways. On 

the one hand, a community provides connotations to a state of being-together, and living side by side in harmony 

with others. On the other hand, Bauman continues, the demands for conformity with the community undermine the 

rights to one’s freedom and self-assertation. The continuous struggle between individuality and the desire to be 

included in the community and sharing a feeling of togetherness with others is by Elias (1991) called “the I/We – 

balance”. This balance could be tilted in either direction depending on the meaning of the negotiated understanding 

of being a member within a certain community. 

If membership in a community is about sharing values, negotiating values and creating shared meaning, it requires 

research to investigate and to try to understand how these processes work in educational settings. In the next section, 

an empirical study is presented aimed at these aspects of membership in the context of an E-OLC, positioned in netbased 

teacher training in Sweden. The analysis shows how the trainees perceive the programme as a venture 

characterized by active participation, collaborative learning, negotiating and sharing of meaning, and some of the 

potential pitfalls in taking theses process for granted. 

The empirical context is a net-based teacher training programme and for that reason, the type of OLC considered 

here is the E-OLC that “…relates to learning activities in schools, colleges and universities. An institution or faculty 

promotes and structures education programs for learners in which the students get credit for what they know and 

what they do.” (Carlén & Jobring, 2005, p. 275). By using tools like blogs, chats and e-mails the members of an E- 

OLC can connect with each other and central features within net-based higher education, like active participation, 

collaboration and dialogue, can take place (Schwier, 2002; Sorensen & Takle, 2004). 

Research Related to the Present Study - A Swedish Perspective 

An increasing amount of research has been carried out in Sweden related to the study presented in this paper. In that 

body of research, it seems possible to identify at least four different themes relevant for the ideas presented in this 

study. One of these themes is related to teacher training. Recent studies conducted by Lindberg (2002) investigate the 

discourses of teacher training and analyze contemporary discussions of teacher training as it has taken shape in 

relation to the present teacher training reform in Sweden. Bernmark-Ottosson (2005) investigates teacher trainees’ 

conceptions of democracy and this is another example of current efforts in this direction. A second theme relates to 

distance education. Thórsteinsdóttir (2005) has investigated information seeking behaviour of distance students and 

Rydberg Fåhræus (2003) has been studying collaborative learning and how this approach can be applied and 

supported in distance education. A third theme is ICT and the internet. Keller (2005) has carried out analyses of 

students’ acceptance of Virtual Learning Environments (VLEs) in a mixed learning environment, and Stigmar (2002) 

investigated how students used the Internet for information seeking. A forth, and final theme relates to research 

aimed at the concept of community. Karlsson (2004) has explored the concept of community in order to gain an 

understanding of how a teacher team functions as a vehicle for the development of competencies in the pedagogical 

use of ICT, and Svensson (2002) uses the concept of community as a theoretical tool in order to understand and 

support IT-mediated communities of distance education. 

The ambition here is not to present a synthesis containing all research conducted in Sweden that can be related to the 

study presented in this paper. Rather, the themes mentioned above are to be understood as attempts to identify what 

already has been researched in Sweden regarding research objects such as teacher training, distance education, ICT 

and the internet, and community. According to Bransford, Brown & Cocking (1999), it is important to conduct 

extensive research on teacher learning, teacher training and how to develop learning opportunities for teachers, in 

order to teach in line with new theories of learning. Further, they claim that research evidence points out that 

successful professional development activity for teachers are extended over time and characterized as encouraging 

for development of teachers' learning communities. They also stress the importance of providing teacher trainees 

with a chance to form teams who stay together in their teacher training. In net-based teacher training such an idea 

30

could be to give the trainees a chance to create an E-OLC that is characterized by active participation, collaboration 

and dialogue. Furthermore, according to Lankshear & Knobel (2006) there is a demand for new ways to train future 

teachers that will face new teaching and learning challenges with young children and students that are growing up 

with mobile devices and new media. To conclude, in net-based teacher training all these ideas and challenges are 

intertwined. It is therefore important to continuously research and develop net-based teacher training programmes if 

teacher trainees of today shall be able to meet the demands from young children and students of tomorrow. 

The Context of the Study 

The context of the study was a net-based teacher training programme provided by a university in northern Sweden. 

This study is a follow-up investigation from a larger case study concentrated on the issue of training teachers through 

technology (Lindberg & Olofsson, 2005). The participants in this study were teacher trainees in the aforementioned 

net-based teacher training programme. The programme investigated was three and a half to four and a half years 

long, depending on the teacher trainees’ choice of degree. The programme was mainly aimed at training teachers to 

be able to teach in sparsely populated areas in Sweden. The programme was mainly conducted via a net-based 

learning environment and only a small number of on-campus gatherings were included. The trainees were divided 

into smaller study groups and the net-based learning environment enabled, throughout the programme, collaboration 

and communication both synchronously and asynchronously without meeting physically. The net-based learning 

environment used in the programme was WebCT and examples of learning tools are email, chat, discussion groups 

with threaded discussions, electronic portfolios, study-group conferences, and students’ café. 

The study group in this paper should be regarded as an E-OLC. The trainees were continuously encouraged by the 

educators to actively use the net-based learning environment. This was for example stressed in the study-guides 

provided for the teacher trainees. The teacher training programme was aimed at educating future teachers with the 

competence of both handling ICT and on using it in order to get in contact with other persons in the surrounding 

society that could enrich their own practice. 

Data Gathering and Procedure 

The total number of teacher trainees enrolled at the time of data gathering was 77. All trainees were asked to be 

interviewed. A group of 22 trainees volunteered, but out of these, three were unable to take part and the interviews 

were carried out with the remaining 19. The trainees that were interviewed were between 20 and 50 years old, 13 of 

the trainees were female and 6 were male. The interviews were semi-structured and each interview was conducted in 

relation to a pre-specified interview guide. The study was conducted in spring 2004. The interview guide was divided 

into 4 different themes, containing 15 questions in total. Before the interviews, each trainee was given an interview 

guide with the intention to allow the trainees to prepare. Each interview lasted for approximately 25 to 60 minutes. 

For this paper, analysis was conducted on data from the theme concerning educational issues. This theme included 

four questions and was constructed with the intention to function as a basis for an open discussion. All answers were 

recorded on tape, transcribed and analyzed. Each interviewee was given the opportunity to comment on the 

transcripts before the analysis. The questions included in the theme were as follows: 

A student in your study group is having difficulties keeping up in the programme and therefore needs help. What 

are your thoughts about being involved in and taking responsibility for the situation? 

There is a new student in the study group. What are your thoughts about being involved and taking responsibility 

for helping the student to feel at ease in the new group? 

What are the important issues that you discuss in the programme? 

What have you learnt in the programme that is especially important for the future? 

Analysis 

The analysis used an approach inspired by hermeneutics (Gadamer, 1976). The approach intended to preserve the 

complexity of the educational setting that was researched. In the analysis, three different categories were constructed 

and presented. The categories emanated from the interpretations of the transcribed data. The categories should not be 

seen as objective truths or as categories capturing actually existing practices within the net-based teacher training 

31

programme. They are rather constructions of the trainees’ discuss aspects of studying, learning and bonding together 

in a programme mostly carried out via the internet. The aim of the analysis is to put forth some aspects of what 

appears to be required from each trainee to be member of an E-OLC in terms of active participation and collaborative 

learning and how this can be understood as a process of negotiating and sharing of meaning with other trainees. 

The categories and the analysis are based on interpretations of the data collected from all four questions included in 

the theme concerned with educational issues. In the process of analysis, each question within the theme has been 

analyzed separately and in the light of the other three questions. The process of interpretation is reflexive in the sense 

that the interpretation of the teacher trainees’ discussions consisted of the interplay between the whole body of data 

collected within the theme and discussion of each trainee. This corresponds to the idea of relating parts to the whole 

and back again in an ongoing circular or spiral of interpretations, all in an order to construct an understanding of the 

data (Gadamer, 1976; Risser, 1997). Further, the interpretations are not to be seen as categorizations into mutually 

exclusive categories but rather a way of describing differences in how teacher trainees viewed participation in the 

programme. Consequently, quotations used in each category can emanate from data collected within all four 

questions included in the theme. The quotations provide specific accounts from the data from which the 

interpretations are made and typical accounts for the trainees’ discussions about being part of a net-based teacher 

training programme. 

Category 1 – The Importance of Contributing to the E-OLC 

In order for teacher trainees in a study group to be able to graduate from the programme, or just pass a single course 

within the programme, they are individually expected to continuously contribute to the learning activities in the E- 

OLC. To the trainees, it appears that they have to be able to adapt to the norms defining how to participate in the 

study group. For instance, this could involve to take an active part in the collaborative process of solving a grouprelated 

task provided by the teacher trainers on the programme. A trainee when discussing co-trainees that do not do 

what is expected expresses this: 

“…if the problem comes from the co-trainee’s lack of pre-knowledge’s, or that they just don’t manage the studies, or 

aren’t able to assimilate their understanding, the question is if we, as a study group should help her or him to pass the 

test. If so, you can question if we in the study group are doing something morally wrong…” 

Furthermore, the contribution should not only agree with what is formulated as a productive contribution in relation 

to, for example, how the task is conceived within the study group, but also when the other members in the group 

expect the contribution to be made. Failure to fulfil what has been negotiated seems often to be referred to a single 

trainee’s lack of individual qualities for studying or lack of self-discipline. When discussing co-trainees not 

delivering what is expected on time, one trainee states that: 

“…I think that you shall take responsibility when you get involved in something, is it [the student’s failure] because 

this person has problems because she or he has 1000 other things besides the studies, I think that you should confront 

them…” 

Additionally, it seems as if the trainees are not interested in acting in the role of a teacher for their co-trainees, but 

rather reflects an understanding implying that each single trainee, and her or his individual contribution, forms the 

collective that together can solve, for example an examination task. Another trainee relates to the same issue by 

saying: 

“…if someone in the [study] group didn’t understand, the others tried to explain. That is something I think works 

well. However, to be the one who tries to help all those who have a problem in understanding could mean that you 

don’t have enough time to manage your own studies and instead, like Florence Nightingale, you flutter around and 

try to help everyone …”. 

The overall understanding within this category seems to be that the valued way to participate in the E-OLC is 

characterized by, for example, individualization, activity, self-regulation and goal-orientation. Being a teacher 

trainee, in terms of membership in an E-OLC in this particular programme, means being part of a collective of 

individuals. 

32

Category 2 – The Importance of Being-Together 

The meanings embedded in this category seems to be in rather strong contrast to category one. Instead of being built 

on an idea in which the individual teacher trainees have to take personal responsibility for their own learning and 

educational success within the programme, another aspect of membership of an E-OLC seems possible to present. 

Following a net-based programme and only having a few physical on-campus gatherings seems to call for another 

way of fulfilling the social dimension in the educational programme. One way of realizing a social dimension 

without meeting face-to-face (f2f) appears to be to use ICT to form an E-OLC. This time, however, it does not 

primarily seem to be the learning process and the goal of graduating from teacher training that is the reason for 

connecting with co-trainees but instead the search for a feeling of being-together. In relation to being separated in 

space and in need of someone to discuss with one trainee says that: 

“…in distance education you are so isolated. The teachers are far away, so you are far away from each other. We 

have learnt to give and take from each other and that we need each other…”. 

There are several trainees that underline the importance of everybody feeling at ease in the group and that it is every 

member’s responsibility in the E-OLC to make this happen. This could be due to the risk that something could 

happen to you and then you would be in need of support within the study group; a study group that mostly is 

accessible via the E-OLC. One trainee expresses such an understanding when stating: 

“Of course I would take responsibility for that [everyone feels welcome in the study group]. Everyone has the duty to 

push each other because you, yourself, could be in a low phase when everything feels difficult. It could be, for 

example, that you have problems at home or something else that makes you need support. It is our [the study 

group’s] mission to support each other”. 

Another aspect of the teacher training programme that seems to require a continuous dialogue between the members 

in the E-OLC is concerning the trainees’ future as teachers. Not specifically how to solve problems related to a 

certain course or task but instead, for example, what courses to chose within the programme and how it will be to 

work as a teacher in “the real world”. One trainee says: 

“…yes, we discuss a lot. Which attitude to have in the classroom together with the pupils and how to handle different 

situations, which working methods to use and of course the salary we will get after completing the education…” 

Another trainee discusses the issue and says: 

“…we discuss study loans, economy of course. How to act in order to get through this training and of course how 

you will work afterwards…” 

An overall interpretation within this second category seems to be that the trainees are in need of the possibility to 

have continuous access to each other. The trainees are located in different geographical settings not able to meet 

physically on a regular basis and therefore their solution is the E-OLC. The contrast with the first category seems to 

be that when discussing issues more related to everyday life or the future life as teachers in school. The norms of 

how to contribute to activities and dialogues within the E-OLC are not so restricted as when contributing to activities 

and dialogues concerning tasks that are more specific and examinations within the programme. 

Category 3 – The Importance of Bridging Distances Between Trainees and Trainers 

It seems that the teacher training programme in question requires continuous contact between both trainee-trainee 

and trainees-trainers. These can be carried out in diverse ways as seen in category 1 and 2 of having dialogues 

concerning different issues related to the programme, and not least to be able to receive response to their completed 

work or to obtain additional perspectives on the discussed issues. It seems, however, that the trainees, more or less, 

lack support from the trainers and instead have to rely on one another and each trainee’s individual capacity to move 

forward within the programme. Sometimes they do not know if they have passed the previously tasks or assignments. 

Students feel the programme lacks support and possibilities to influence communication, which are required in order 

to achieve a secure and improving teacher training in collaboration with the staff at the university. The norm seems 

33

to be that each individual trainee must take responsibility and be motivated to keep up their studies even if they 

receive no feed-back or support from the trainers. One trainee expresses this by saying: 

“…they [the trainers] just wait, and wait, and because I haven’t received anything back I assume that I have failed 

the test… The same thing happens when we post questions on the net, and it takes three weeks before we get any 

answers, or even worse when we don’t receive any answers at all, and the trainers says that they have seen us 

discussing these questions on the net… I hope that I will remember that feeling when I teach my pupils in school and 

when we talk about democracy. It [democracy] is non-existing in the university…” 

Furthermore, it seems as if problems needing to be solved and questions are related to when the trainees attending 

the university physically. The trainees put forth that they use the E-OLC to support each other between on-campus 

gatherings and, as one trainee puts it, that: 

“…we write e-mails to each other and when, during on-campus gatherings, we try to straighten things out…” 

Another aspect of the importance of bridging distances between trainees and trainers relates to how the on-campus 

gatherings are carried out. The gatherings are mostly built up around lectures and seminars during which different 

content matters are problematized. The analysis suggests that trainees would like more social activities during oncampus 

gatherings, which would provide opportunities for bonding and strengthening their relationships. These 

relationships could thereafter continue to be strengthened via the E-OLC. Additionally, there seems to be a wish for 

more occasions for trainees and trainers to collaboratively discus the complex task of being a teacher. One trainee 

states: 

“…a thing that I have thought a lot about the social dimension in the teacher profession. We are becoming so much 

more than just a person that has to transfer blocks of knowledge’s to them [the pupils]. We have to actually be there 

and foster them and give them [the pupils] values and every thing. We think that many damn fine words have been 

written, but there is a great distance to the pupils. It should be so much better if some time was spent on trainers 

connecting with the trainees instead of writing fancy papers…” 

Within this third category, the overall understanding possible to put forth, is that there is a lack of continuous contact 

between trainees and trainers. For example, a dialogue in which the two parts could negotiate around issues like how 

to meet the pupils in the classroom and the meaning of the constitutive values underpinning the Swedish school 

system. Furthermore, it seems as if trainers, to some extent, reject using ICT in order to report results from different 

assignments and refrain from joining the discussion taking place within the E-OLC. The trainees put forth that they 

want a social dimension in the programme, despite their separation in space, but that the staff at the university does 

not appear to share this progress. 

Discussion 

The net-based teacher training programme studied in this paper seems to be of a complex nature and it seems to be 

difficult to be a successful participant. When contrasting the three categories described in the analysis, three issues 

supporting this statement became particularly apparent. The first issue suggests that the individual trainee is held 

responsible for their learning. Success appears to be due to each trainee’s capacity to contribute to the collaborative 

work conducted within the study group. Failure, though, seems to be just around the corner for a trainee who has 

problems living up to this negotiated norm. It appears to be difficult to trust the other members in the E-OLC, and to 

act as a united community aimed at solving different kind of educational problems or tasks through a joint process of 

collaboration. Provocatively formulated, if you do not contribute with something or participate in line with what is 

expected, you had better not contribute or not participate at all! 

The second issue suggested how the individual trainee needs others. There seems to be a fear among the teacher 

trainees, a fear of being the next one with problems and to be in the hands of the cyber space in order to get in 

contact with co-trainees. This fear seems to promote a more inclusive attitude to other members and how they 

participate and express their ideas within the E-OLC. Being-together in the E-OLC is important, which is in 

opposition to the embedded meaning in the first issue. 

34

The third issue suggested difficulties in bridging the distance between trainees-trainees and trainees-trainers. This 

seems strange, as ICT is the tool to use in order to connect the trainees to the university. This seems to reflect a view 

that the trainees are supposed to manage on their own in the programme. The trajectory discussion very much 

becomes a question of the trainees’ prejudices of what to expect after graduating and how to handle morally and 

ethically complex situations when working as teachers. This result is interesting also in the light of a study made by 

the Swedish Knowledge Foundation (2005) reporting on teacher trainees’ attitudes towards ICT. Teacher trainees 

stress that teacher trainers’ ICT competence in education and teaching is poor and that their knowledge of ICT as a 

pedagogical tool is poor. A majority of the teacher trainees further state that they are positive towards using ICT and 

the internet in their teaching and that they will use ICT as a pedagogical tool. 

What can we learn from the study presented in this paper in order to support learners in this kind of educational 

programme? It seems possible to claim that a pedagogy relying on being designed for participation, in this case 

neglects that individuals may need their individual design (as well). The lack of collective solutions to satisfy 

trainees’ needs for others outside the study group may restrict their access to the educational content. Furthermore, 

the designed participation restricts the trainees’ studies, restricting them to wait for trainers to respond, and fellow 

trainees to react to their needs. Conclusively, if viewed as a pedagogical idea for higher education, designing E- 

OLCs might bring severe difficulties about. 

On the other hand, it seems possible to claim that the need for socializing and support, both from fellow trainees and 

from trainers, suggests that the E-OLC has an important role to fulfil. The aspects of negotiation of meanings and 

values to underpin the education in question are not to be taken for granted or merely handed over to the trainees 

without support. In order to support such a development in net-based teacher training programmes, more research 

seems to be required to construct an understanding of how the processes of participation and collaboration can be 

brought about, and at the same time lay an important foundation for processes of negotiating of meaning where the 

ethical aspects of being-together are in focus. 

The educational visions of the Bologna process and policies of the European Union will affect the way teachers are 

trained in the future. The next step in higher education today seems to build individual knowledge and at the same 

time build a future citizenship; a membership in Europe. In this process, teachers of tomorrow will play a crucial role 

for such a development. However, the question remains how people will react to the intentions of being designed to 

become the Europeans of tomorrow. Will belonging to the future European community of learners be tempting, will 

it offer such safety, so people are willing to give up parts of their sovereignty and become what they are expected to 

become? To what extent and towards what ends are the I/We balance (Elias, 1991) to be tilted? In each of the three 

categories described, different aspects are in focus and different rationales seem to be working. 

Conclusions 

When being part of an educational experience, in this case being part of a net-based teacher training programme, 

students transcend the intended use of both ICT and design. A pedagogy in which the being-together is the starting 

point seems to be called for when designing this kind of educational programme. E-OLCs need to be designed with 

the pedagogical issues firmly based in an ethical view of people and education. Having to rely on each other in the 

study-group situated within an E-OLC, simply because this is the designed structure to support the studies, becomes 

part of a process of negotiation in which the trainees seems left to their own devices. What is negotiated can hardly 

be designed, it seems. Therefore, it might well be that access to education (the intended way, based on participation, 

collaboration and negotiation) are denied some trainees if they do not adhere to the negotiated ways. I conclude this 

paper by emphasizing the need of a pedagogical approach that relies heavily on social, collaborative and ethical 

aspects of learning as a starting point when designing the E-OLCs to support the kind of education we in Sweden as 

well as EU need for the XXI century. 

References 

Bauman, Z. (2001). Community - Seeking safety in an insecure world, Cambridge: Polity Press. 

Berg, G. A. (2003). The knowledge medium: designing effective computer-based learning environments, Hershey, 

PA: Information Science Publishing. 

35

Bernmark-Ottosson, A. (2005). Demokratins stöttepelare: en studie av lärarstuderandes demokratiuppfattningar 

[Pillars of democracy. A study of teacher students’ perceptions of democracy], Doctoral thesis, Karlstad: 

Avdelningen för pedagogik, Institutionen för utbildningsvetenskap, Karlstads universitet. 

Bonk, J.C., & Cunningham, D.J. (1998). Searching for learner-centered, constructivist, and sociocultural components 

of collaborative educational learning tools. In Bonk, C.J. & King, K.S. (Eds.), Electronic Collaborators. Learner- 

Centered Technologies for Literacy, Apprenticeship, and Discourse, Mahwah, NJ: Lawrence Erlbaum, 25-50. 

Bransford, J. D., Brown, A. L., & Cocking, R. R. (1999). How People Learn: Brain, Mind, Experience, and School, 

Washington, D.C.: National Academy Press. 

Carlén, U., & Jobring, O. (2005). The rationale of online learning communities. International journal of web based 

communities, 1 (3), 272-295. 

Elias, N. (1991). The Society of Individuals, London: Continuum. 

Etzioni, A. (1993). The Spirit of Community. Rights, Responsibilities and the Communitarian Agenda, London: 

Fontana Press. 

EU legislation (2003a). eLearning - Designing tomorrow’s education, retrieved October 7, 2007, from, 

http://europa.eu.int/scadplus/scad_en.htm. 

EU legislation (2003b). Concrete future objectives of education systems, retrieved October 7, 2007, from, 

http://europa.eu.int/scadplus/leg/en/cha/c11049.htm. 

Haythornthwaite, C., Kazmer, M.M., Robins, J., & Shoemaker, S. (2000). Community development among distance 

learners: Temporal and technological dimensions. Journal of Computer-Mediated Communication, 6(1), retrieved 

October 7, 2007, from, http://jcmc.indiana.edu/vol6/issue1/haythornthwaite.html. 

Howard, C., Schenk, K., & Discenza, R. (2004). Distance learning and university effectiveness: Changing 

educational paradigms for online learning, London: Information Science Publishing. 

Jaldemark, J., Lindberg, J.O., & Olofsson, A.D. (2005). Att förstå hur man deltar via redskap i en lärgemenskap. In 

Jobring, O. & Carlén, U. (Eds.), Att förstå lärgemenskaper och mötesplatser på nätet [To understand learning 

communities and places to meet on the net], Lund: Studentlitteratur, 109-147. 

Karlsson, M. (2004). An ITiS Teacher Team as a Community of Practice, Doctoral thesis, Göteborg: Acta 

Universitatis Gothoburgensis. 

Keller, C. (2005). Virtual learning environments in higher education: a study of students' acceptance of educational 

technology, Doctoral thesis, Linköping: Department of Computer and Information Science, Linköpings University. 

Lankshear, C., & Knobel, M. (2006). New Literacies: Everyday Practices and Classroom Learning, Berkshire, UK: 

Open University Press. 

Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation, Cambridge: Cambridge 

University Press. 

Lewis, D., & Allan, B. (2005). Virtual learning communities. A guide for practitioners, Berkshire, UK: Open 


Lindberg, J.O., & Olofsson, A.D. (2005). Training Teachers Through Technology. A case study of a distance-based 

teacher training programme, Doctoral thesis, Umeå: Umeå University. 

36

Lindberg, J.O., & Olofsson, A.D. (2006). Distancing Democracy: organising on-line teacher training to promote 

community values. UCFV Research Review, 1 (1), retrieved October 7, 2007, from 

http://journals.ucfv.ca/rr/RR11/article-PDFs/lindberg-olofsson.pdf. 

Lindberg, J.O., & Olofsson, A.D. (in press). OLC in the context of the Other – Face, trace and cyberspace. 

International Journal of Web Based Communities. 

Lindberg, O. (2002). Talet om lärarutbildning [Teacher Education: How we talk about it and what it means], 

Doctoral thesis, Örebro: Universitets-biblioteket. 

Lock, J. V. (2002). Laying the groundwork for the development of learning communities within online courses. 

Quarterly Review of Distance Education, 3 (4), 395-408. 

Olofsson, A.D. & Lindberg, J.O. (2005). Assumptions About Participating in Teacher Education Through the Use of 

ICT. Campus Wide Information Systems, 22 (3), 154-161. 

Olofsson, A.D., & Lindberg, J.O. (2006). Enhancing Phronesis. Bridging Communities through Technology. In 

Sorensen, E. K. & Murchú, D. Ó (Eds.), Enhancing Learning Through Technology, London: Information Science 

Publishing, 29-55. 

Palloff, R.M., & Pratt, K. (2005). Collaborating online. Learning together in community, San Francisco, CA: Jossey- 

Bass. 

Risser, J. (1997). Hermeneutics and the voice of the others. Re-reading Gadamer´s Philosophical hermeneutic, New 

York: State University of New York Press. 

Rydberg Fåhræus, E. (2003). A triple helix of learning processes: how to cultivate learning, communication and 

collaboration among distance-education learners, Doctoral thesis, Stockholm: Stockholms universitet. 

Schwier, R. A. (2002). Shaping the Metaphor of Community in Online Learning Environments. Paper presented at 

the International Symposium on Educational Conferencing, June 1, 2002, Banff, Canada, retrieved October 7, 2007, 

from http://cde.athabascau.ca/ISEC2002/papers/schwier.pdf. 

Sergiovanni, T. (1999). The story of community. In Retallick, J., Cocklin, B. & Coombe, K. (Eds.), Learning 

Communities in Education, London: Routledge, 9-25. 

Seufert, S., Lechner, U., & Stanoevska, U. (2002). A reference model for online learning communities. International 

journal on E-learning, 1 (1), 43-55. 

Sorensen, E.K., & Ó Murchú, D. (2004). Designing Online Learning Communities of Practice: A Democratic 

Perspective. Journal of Educational Multimedia, 29 (3), 189-200. 

Sorensen, E.K., & Takle, E.S. (2004). A cross-cultural cadence. Knowledge building with networked communities 

across disciplines and cultures. In Brown, A. & Davis, N. (Eds.), World Yearbook of Education 2004. Digital 

Technology, Communities & Education, London: Routledge Falmer, 251-263. 

Stigmar, M. (2002). Metakognition och Internet. En undersökning om gymnasieelvers informationsanvändning 

[Metacognition and the Internet. A study of high-school students´ use of information], Doctoral thesis, Växjö: Växjö 


Svensson, L. (2002). Communities of Distance Education, Doctoral thesis, Göteborg: Göteborgs universitet. 

Söderström, T., Hamilton, D., Dahlgren, E., & Hult, A. (2006). Premises, promises: connection, community and 

communion in online education, Discourse, 27 (4), 533-549. 

37

The Knowledge Foundation (2005). IT och lärarstuderande 2005 [IT and Teacher Trainees 2005], Stockholm: 

Stiftelsen för kunskaps- och kompetensutveckling. 

The Swedish National Agency for Higher Education (2005). A follow-up on the Swedish Net University. Final report 

2: Accessibility, recruitment and extra compensation, Report No. 49, Stockholm: The Swedish National Agency for 

Higher Education. 

Thórsteinsdóttir, G. (2005). The information seeking behaviour of distance students: a study of twenty Swedish 

library and information science students, Doctoral thesis, Göteborg: Inst. f. biblioteks- och informationsvetenskap, 

Göteborgs universitet. 

Wenger, E. (1998). Communities of practice. Learning, meaning and identity, New York: Cambridge University 

Press. 

38

Svensson, L., & Östlund, C. (2007). Framing Work-Integrated e-Learning with Techno-Pedagogical Genres. Educational 


Framing Work-Integrated e-Learning with Techno-Pedagogical Genres 

Lars Svensson 

Forum for Work-Integrated Learning, University West, Sweden // lars.svensson@hv.se // Tel: +46 733 975133 

Christian Östlund 

Laboratory for Interaction Technology, University West, Sweden // christian.ostlund@hv.se // Tel: +46 520 223567 

ABSTRACT 

Distance Educational Practice is today supported by a range of information systems (IS) design theories. Still, 

there are surprisingly few strong pedagogical ideas and constructs that are communicated across distance 

educational institutions. Instead it is often the technology, the software and the medium that is at the centre of 

attention as we frequently discuss notions such as learning management systems, courseware, chat room, 

streaming media and blogs. This paper argues that design concepts should be used to bridge the gap between 

design theories and distance educational practice. It is also argued that genre theory could be instrumental in 

framing the characteristics of such techno-pedagogical genres in a way that constitutes a powerful level of 

communicating and disseminating new ideas within and across educational communities. 

Keywords 

Genre, IS design theory, Design concept, Techno-Pedagogical Genre 

Introduction 

Looking back on the last decades of development within the field of distance education reveals a rich spectrum of 

initiatives that has been implemented and evaluated in educational as well as organisational settings. It is probably 

fair to state that the extent to which these initiatives and innovations are related to, or derived from, theory varies to a 

large degree. In an influential paper from 2002, Markus, Majchrzak and Gasser (Markus et al., 2002) presented a 

model for IS design theory (based on the work of Walls (1992)), where they stated that a design framework should 

be firmly rooted in a kernel theory that guides the elicitation of requirements, consequently transformed into 

principles for design and development. Examples of work that use this (or similar) approaches to design for learning 

and competence development are Herrington & Oliver (1995), Hung & Chen (2001), and Hardless (2005). Applying 

the approach suggested by Markus et al., (2002) is likely to generate design frameworks (i.e. theories) that are 

theoretically sound. Still, a central problem is to bridge a generic design theory with the specific contexts and content 

where theory is to be applied in actual design practice. Hardless (2005) suggest that design concepts could be 

instrumental to that effect. 

“Design concepts served the role as an intermediate conceptualization between design theory and 

concrete prototype. A design concept is here a collection of general ideas and principles for a type of 

CDS [Competence development system]. In other words, a definition of a particular type of learning 

intervention abstracted beyond specific instances or realizations based on the design concept.” 

(Hardless, 2005) 

Hence, design concepts are to be derived from, and evaluated against the design framework where the kernel theory 

(fig 1:1) generates requirements for the phenomenon design is intended to support and develop. Requirements (fig 

1:2) are then transformed into design guidelines and design concepts (fig 1:3) are derived from the design theory. 

Design concepts (fig 1:4) are realized into prototypes and systems in various practices and the system evaluation (fig 

1:5) feeds back to design concepts and design theory. In the context of e-learning we believe there is a need for this 

type of intermediary level of abstraction in between educational practice and IS design theory. However, the question 

of how to describe and frame a design concept still remains. In this paper we argue that the notion of genre is a 

promising approach in framing new and innovative techno-pedagogical ideas, and provide a structured approach to 

how such ideas should be deconstructed and presented. The ideas are illustrated through the presentation of three 

simple design concepts for work-integrated e-learning: (i) Web Lecture, (ii) Blog Reflection, and (iii) Competence 

Kick-off. 






39

Figure 1. Design Theories in context (Inspired by Markus et al (2002) and Hardless (2005)). 

The next section briefly presents IS design theories for learning. This is followed by a section that outlines the 

fundamental concepts of Genre Theory. In section four the framework for techno-pedagogical genres is presented 

together with three illustrating examples. The paper concludes with a discussion of the relationship between 

intentional design and actual educational practice. 

IS Design Theory for Learning 

There is a growing body of research addressing various aspects of online learning design with focus on i.e. student 

centered learning (Uskov, 2004), reusability of learning resources (Uskov, 2004; Wills et al., 2002; Aroyo & 

Dicheva, 2004) team approach for iterative re-design and maintenance on educational on-line resources (Sims & 

Jones, 2002) strategies to enhance student motivation, confidence and control (Astleitner & Hufnagl, 2003), problem 

based learning (Slough et al., 2004) and quality assurance (Bohl et al., 2002). 

According to Herrington and Herrington (2006), based on a paper by Herrington and Oliver (1995), online learning 

environments can and should use situated learning as an approach to design. The elements that should be 

incorporated into the design in order to achieve what Herrington and Herrington (2006) refer to as authentic learning 

are authentic context, authentic activities, access to expert performances, multiple roles & perspectives, 

collaboration, reflection, articulation, coaching & scaffolding and lastly authentic assessment. The context is 

authentic when it is all-embracing and provides the purpose for learning, simply providing examples from real-world 

situations is consequently not enough. The tasks and activities the students perform should be ill-defined but still 

with real-world relevance and be completed over a sustained period of time, rather than many short and disconnected 

examples. Access to expert performances is achieved by giving the students a model of how a real practitioner 

behaves in a real situation, e.g. case-based learning or video showing expert performance. Students should be 

encouraged to explore multiple roles & perspectives’, providing one “correct” view is not false but inadequate. 

Collaboration needs to be designed to engage higher-order thinking so the collaboration between learners requires 

them to predict and hypothesize, and then suggest a solution. Along the same line is reflection upon a broad base of 

knowledge supposed to be encouraged. The environment should also ensure that the learners work and discuss in 

groups, present their findings in order for them to articulate, negotiate and defend their knowledge. An authentic 

learning environment should accommodate a coaching & scaffolding role for the teacher and not just a didactic role 

telling students what they need to know. The students can also assume a coaching & scaffolding role if a 

collaborative environment is provided. Authentic assessment needs to be integrated with the learning activity. It does 

not necessarily have to be done by conventional methods such as examination and essays, but can be in the form of 

statistics of the learner’s path through multimedia programs, diagnosis or reflection and self-assessment. (Herrington 

& Herrington, 2006) 

40

Hung and Chen’s (2001) design framework identifies four principles of learning (from a situated learning and 

Vygotskian view) and derives from these four design considerations for e-learning: Situatedness i.e. e-learning 

environments should be Internet based so that learners can access the learning environment in their situated contexts 

and thereby being able to focus on tasks and projects, thus enabling learning through doing and reflection-in-action. 

Commonality, i.e. e-learning environments should create a situation where there is continual interest and interaction 

through the tools and capitalize on the social communicative and collaborative dimensions. E-learning environments 

should also have scaffolding structures which contain the genres and common expressions used by the community. 

Interdependency i.e. e-learning environments should create interdependencies between individuals where novices 

need more capable peers capitalizing on the zone of proximal development and the diverse expertise in the 

community. E-learning environments should be made personalized, with tasks that are meaningful to the learner in 

their context and personalized strategies and content by tracking the learner’s history, profile, and progress. 

Infrastructure i.e. e-learning environments should have structures and mechanisms set up to facilitate, in a flexible 

way (anywhere and anytime), projects where learners’ are engaged in. 

Genre theory 

To some extent genre is an intuitive concept that points to typified and recognizable features of a phenomenon, but 

the notion of Genre has also been frequently used as a theoretical construct for various purposes, and within different 

theoretical traditions (Roberts, 1998; Ryan et al., 2002; Svensson, 2002; Shepherd et al., 2004; Saebø & Päivärinta, 

2005). The primary strength of genres seems to be to provide tools to describe a phenomenon, but when resting on 

supporting theories such as for instance structuration theory or social theories of learning the scope sometimes 

stretches to understand the processes that put the genres in play. Orlikowski and Yates (1994) focus on how 

electronic interaction within an organisational community is typified into genres with characteristic purpose, 

structure and form. A genre is defined as: 

Typified communicative act having a socially defined and recognised communicative purpose with 

regard to its audience (Orlikowski & Yates, 1994). 

In addition to the definition, Orlikowski and Yates (1994) make several clarifications that are helpful in order to 

grasp the genre concept. Firstly, they emphasise that the purpose referred to in the definition should be recognised 

and shared within the community/group/organisation, and not to be interpreted as the purpose of individual 

community members. Secondly, they state that a stable substance and form should be connected to such a shared 

purpose in order to constitute a genre. Substance refers to the topics and the discursive structure of the interaction, 

and form has three sub-dimensions: structural features, communicative medium and language. In a similar manner, 

Shepherd and Watters (1998) define a Cybergenre to be characterised by content, form and functionality. 

Furthermore, Orlikowski and Yates (1994) states that a collection of genres can constitute a Genre Repertoire, i.e. the 

complete set of genres used for interaction within a community. They say: 

Both the composition as well as the frequency with which they are used are important aspects of a 

repertoire. When genres are heavily intertwined and overlapping it may be useful to talk of a genre 

system, where genres are enacted in a certain sequence with interdependent purpose and form 

(Orlikowski & Yates, 1994). 

Similar to the way Wenger (1998) sees the shared repertoire of a physical community as an important element in the 

definition of that community, Orlikowski and Yates (1994) underscores how electronic communication genres 

frames and indirectly defines the community space by serving as a "social template" for work. However, it is 

important not to perceive genres as inherently static. Instead, genres evolve over time, partly as a result of technology 

adaptation and innovation (Shepherd & Watters, 1998), and partly as a result of community negotiations (Wenger, 

1998). 

Since genres, represented by simple labels such as “western movie”, “mystery novel” or “academic seminar”, carry 

so much detailed information of form, content and functionality for the audiences or communities that are familiar 

with the genre, they could also be interesting with respect to design. In the context of distance educational practice it 

is surprising to notice how few strong pedagogical ideas and constructs that are communicated across distance 

educational institutions. Instead it is often the technology, the software and the medium that is at the centre of 

41

attention as we frequently discuss notions such as learning management systems, courseware, chat room, streaming 

media and blogs. At the same time we can easily find many examples of stable educational genres in traditional (non 

technology mediated) education. Concepts such as lectures and seminars dates back to the very beginning of schools 

and education, but also more novel examples such as multiple-choice exam or term paper are widely recognized with 

respect to form and structure across institutions and countries. 

A genre framework for techno-pedagogical design concepts 

This section outlines a tentative framework for description of techno-pedagogical genres. The framework is 

organised as a table with three rows – one for each of the primary genre dimensions (form, content and 

functionality). The framework uses the narrative structure, i.e. a series of activities organized, sequentially or 

parallel, in time as the primary structural element. The narrative, being an observable structural feature obviously 

relates to the form of the genre. Each activity in the narrative generates a column in the framework, where 

information on content and functionality can be added. Figure 2 shows how the narrative structure can be illustrated 

graphically. The symbols used to illustrate the narrative structure also capture additional form elements. Rectangles 

imply information activities, whereas circles relate to communicate activities, and by tilting a rectangle 45 degree it 

is possible to differ between student-centered and teacher-centered information activities. A specific symbol is also 

used to mark the position of a synchronization point (Lundin, 2003), i.e. a point in time in the sequence of activities 

where participants need to be temporally coordinated (typically a deadline). 

Synchronization 

Point 

Figure 2. Graphical illustration of the symbols used to describe the narrative structure of techno-pedagogical genres 

In the following sub-sections three different techno-pedagogical genres that has been developed in connection with 

various design-oriented research projects at Laboratorium for Interaction Technology (University West, Sweden) is 

presented in order to highlight how the framework can be used to describe a techno-pedagogical genre. These design 

concepts have not been chosen on merits of innovation and novelty, but rather as examples where IS-design and 

instructional design are integrated. 

Web Lecture 

Teacher 

Information 

Activity 

Student 

Information 

Activity 

Communication 

Activity 

Group 

The web lecture design concept was developed through a grounded approach where existing “web lecture practice” 

in four different courses were examined. The study identified central design elements which subsequently were 

evaluated through a student survey and through comparison with design theories (Hung & Chen, 2001; Herrington & 

Oliver, 1995) (see Svensson & Östlund, 2005 for further details). 

The first activity is when a lecture guide document is published on the course web-site. The document typically 

contains student support for engaging with central concepts, and supplements to text books such as examples, 

exercises etc. Shortly after the lecture notes (powerpoint-file) are published on the web, and can be used for lecture 

preparations and as a basis for more detailed lecture notes. The lecture is a student controlled video-stream, typically 

consisting of three frames (1) Video of teacher, (2) animated slides, and (3) interactive table of content (fig 3). One 

of the closing modules of the video lecture typically specifies the assignment that the students should work on, 

relating to the topic of the web lecture. Assignment-work is coached and tutored using a discussion forum and the 

module closes with the students getting feedback on their submitted work. In table 1, the web lecture genre described 

above is deconstructed into a genre framework matrix with three rows corresponding to the genre dimension of form, 

content and functionality. 

Work 

Parallel 

Activities 

42

Narrative 

Structure 

(Form) 

Content 

(Substance) 

Functionality 

(Medium) 

Figure 3. Screen dump from web lecture on “How to search the internet” 

Study Guide and 

Lecture Slides 

containing: 

Presentation of central 

concepts. 

Supplements to lecture 

readings. 

Examples and 

exercises. 

Support for lecture 

notes 

Pre-Lecture 

Material 

Text, graphs and 

hyperlinks to web 

resources 

Electronic document 

with text, images and 

graphs 

Table1. Genre framework for Web Lecture 

Video 

Lecture 

Modularized 

information of 

subject matter. 

Presentation of 

Students’ 

assignment(s) 

Hyperlinked 

table of content. 

Student control 

of video-stream 

(fast forward, 

rewind, pause 

etc.) 

Coaching 

T 

Student-student and 

student-instructor 

interaction 

Text-based 

interaction in 

threaded discussion 

forum or multiparty 

video-conference 

session 

Student 

Paper(s) 

Students’ 

Individual paper 

Electronic 

document with 

text, images and 

graphs 

Deadline 

Feedback 

T 

Teacher’s 

individual 

feedback to 

students 

Anchored 

annotations in 

electronic 

document 

43

Blog Reflection 

This design concept was developed in a Ph.d.-course on work-integrated learning with participants from several 

disciplines with in social sciences. The narrative structure depicted below was repeated four times (in four 

consecutive modules). The core IT-support for this simplistic design was a standard blog system (fig 4). 

Figure 4. Standard system used for the Blog Reflection 

The overall purpose is for students to reflect and discuss on a common reading assignment. The first activity is 

streamed video-vignette which introduces and problematize central themes, concepts and theories from the literature. 

This is followed by individual reading in parallel with joint discussion on the web. Each student is then expected to 

publish a brief text where she relates the literature to her own dissertation project. The reflections are posted as 

entries to the blog. Each reflection receives commentary from two other students, and the module is closed by the 

teacher who writes a meta-reflection on all contributions. Table 2, outlines schematically how the blog reflection 

genre can be described using the genre framework matrix. 

Narrative 

Structure 

(Form) 

Content 

(substance) 

Functionality 

(Medium) 

Vignette 

Introduction of 

reading 

assignment 

Presentation of 

central 

concepts and 

theories 

Video-stream 

with student 

control 

Table 2. Genre framework for Blog reflection 

Joint 

Discusssion 

Student-student 

discussion 

parallel to 

individual 

reading 

Threaded textbased 

discussion 

forum 

Publish 

Reflections 

Deadline 

Short personal 

reflection on the 

literature 

Text-based Blog 

entry 

Writing 

Rreviews 

Publish 

Reviews 

Deadline 

Comments and reviews of 

at least two other entries 

Teacher 

Wrap Up 

Teacher’s 

metacomments 

on 

all reflections 

and reviews 

Standard Blog Follow-up Blog entry 

44

Competence Kick-off 

The competence kick-off was developed in an action research project together with a network organization 

consisting of SME in the automotive sector of Sweden. The design concept was developed together with a group of 

practitioners, and was a response to the fact that many of the participating practitioners expressed frustration over 

past experiences when having used external experts for introductions to new knowledge domains. The frustration 

related firstly to a mismatch between what was expected and what was delivered by the expert, or as expressed by 

one of the practitioners: 

“Most times the so called experts fail to give you what you really need. Either their talk is too basic or 

they are far to advanced” (HR Manager) 

Secondly, many of the practitioners stated that they often felt that the audience of an expert lecture or seminar was 

ill-prepared and consequently, it was often difficult to see that these activities left any trace in the organization. To 

address these shortcomings a design concept with six distinct activities were developed. The Competence Kick-Off 

genre is to some extent inspired by the pedagogical traditions of liberal adult education (Hawke, 1999), where “good 

conversations” in “learning circles” are central elements. In many cases, learning circles lacks the traditional 

teacher/expert. Instead participants are more or less equally novice to the subject, and goals and objectives are 

discussed among the participants, of which one act as a circle-leader or a moderator. In the first step the title of the 

competence kick-off is set and advertised in the network. Participants enrol, and an expert is contacted and 

contracted. After having studied some introductory material, the students gather for a moderated discussion seminar 

where the goals and knowledge levels of each participant is presented, discussed and negotiated resulting in a jointly 

agreed upon requirement specification that clearly states what themes and questions the expert should address. After 

negotiating the requirements with the expert, an interactive expert presentation is performed. Subsequently, the 

participants meet to evaluate and discuss the outcome, and how to proceed. Finally, the experiences are documented 

in a white-paper that could be distributed to other interested parties in the network 

. 

Figure 5. Mindmap tool for phase two of competence kick-off 

The Competence kick-off is at present being evaluated in three different settings with three different themes (“Digital 

Video” for a group of journalists, “Geographical Information Systems” for a network of public administrators, and 

“Organizational Culture” for SME managers). The evaluations have generated several implications for design 

regarding for instance online-templates for the requirement specification document and the white paper, as well as a 

mind-mapping tool for framing, deconstructing and prioritizing the theme (fig 5). The current synthesis of design 

implications for thecompetence kick-off genre is summarized in the genre framework of table 3. 

45

Narrative 

Structure 

(Form) 

Content 

(Substance) 

Functionality 

(Medium) 

Conclusion 

Introducing 

The Theme 

A title is set 

for the circle, 

and an expert 

is contracted. 

Participants 

work with 

introductory 

material 

Introductory 

texts and/or 

video clips 

Table 3. Genre framework for Competence Kick-off 

Framing and 

Prioritizing 

Participants 

discuss & 

negotiate 

their 

objectives 

and needs. 

Thereby 

identifying & 

prioritizing 

core 

questions and 

themes 

Computer 

Conference 

System or 

Video 

Conference 

Graphical 

tool for mindmapping 

Requirement 

Spec. 

Document 

A formal 

document that 

is negotiated 

with the 

expert 

regulating 

content and 

goals for the 

expert 

presentation 

Collaborative 

authoring tool 

or Blog 

Expert 

Presentation 

Interactive 

seminar 

Streaming 

video 

suplemented 

with tool for 

synchronous 

text-communication 

De-Briefing 

Group 

discussions 

following up 

& evaluating 

the expert 

seminar. 

Threaded 

discussion 

forum or 

videoconference 

A jointly 

authored 

document 

describing the 

outcome of 

the 

competence 

kick-off. 

Intended also 

for sharing 

with non- 

participants. 

Collaborative 

authoring tool 

or Blog 

This paper wants to make a fairly simple point with respect to design of educational technology, innovation and 

dissemination of techno-pedagogical ideas need design concepts that can bridge the gap between highly abstracted 

IS design theories for learning and the situated nature of e-learning practices. We have presented concrete examples 

where techno-pedagogical design concepts, framed by the genre dimensions form, content and functionality, are 

instrumental to that effect. 

We think there are primarily two major merits from using a genre approach to e-learning design Firstly, we think that 

the structure dimension of a genre captures the inherently narrative property that is central for any educational 

design, and by viewing the design as a series of activities that unfolds as a narrative or a story we can capitalize on 

the communicative strengths of narratives that can be effective as agents for innovation and change (Bolin et al., 

2006) 

Secondly, the interplay between structure, content and functionality has the potential of letting the designer work 

with an integrated design rationale rather than with separate design agendas for instruction and technology (Svensson 

& Östlund, 2005) 

To back-up these claims we are drawing from the experiences of the collaborative design work where the design 

concepts presented in this paper have been realised into systems and systems prototypes. These experiences and the 

evaluations of the systems further stresses that the genre framework functioned well as a tool for collaborative and 

multi-disciplinary design. Furthermore, given the fact that the Competence kick-off genre has been applied in three 

different contexts, with varying content and audience suggests that the level of abstraction of a techno-pedagogical 

genre makes it suitable for flexible translations. 

However, it is important to acknowledge that the proposed framework for techno-pedagogical genres does not 

replace the need for design theories for various learning contexts. Instead genres should be carefully developed in 

White 

Paper 

46

close dialogue with theory. Furthermore, the level of description with respect to IS support provided by the 

functionality element of the framework can hardly be fine-tuned enough to capture all the requirements and 

specification of the systems and applications needed to realise a certain instantiation of a genre, a fact that stresses 

the importance of design theories that explicitly guides design choices and development strategies. 

Finally, it must be stressed that acts of intentional design can not fully determine what will constitute a genre once it 

is put in to use. Intentional design can at best suggest a “scope of action” or a “social action space”, where some 

actions are encouraged and supported and others are obstructed and discouraged (Löwgren & Stolterman, 2004; 

Köhler, 2006). In other words - the full flavour of an educational genre is negotiated and enacted within a situated 

community of participants. Hopefully, over time, strong and successful genres will be designed, communicated, 

disseminated and enacted across educational cultures. 

Acknowledgement 

This work has partly been sponsored by the European Union Structure Fund (Area 2), and partly by the Swedish 

Knowledge and Competence Foundation (Learn-IT program). 

References 

Aroyo, L., & Dicheva, D. (2004). The New Challenges for E-learning: The Educational Semantic Web. Educational 

Technology & Society, 7 (4), 59-69. 

Astleitner, H., & Hufnagl, M. (2003). The Effects of Situation-Outcome-Expectancies and of ARCS-Strategies on 

Self-Regulated Learning with Web-Lectures. Educational Multimedia and Hypermedia, 12 (4), 361-376. 

Bohl, O., Winand, U., & Schellhase, J. (2002). A Conceptual Framework for the Development of WBT-Guidelines. 

Paper presented at the E-Learn 2002, 15-19 October, 2002, Montreal, Canada. 

Bolin, M., Bergqvist, M., & Ljungberg, J. (2004). A Narrative Mode of Change Management. In Flensburg, P. & 

Ihlström, C. (Eds.), Proceedings of IRIS27, Falkenberg, Sweden. 

Hardless, C. (2005). Designing Competence Development Systems, Doctorial dissertation, Department of 

Informatics, Gothenburg University, Sweden. 

Herrington, J., & Oliver, R. (1995). Critical Characteristics of Situated Learning: Implications for the Instructional 

Design of Multimedia. In Pearce, J. & Ellis, A. (Eds.), Learning With Technology, Parkville, Victoria: University of 

Melbourne, 235-262. 

Hawke, B. (1999) Adult Education Research Trends in the Western European Countries. In W. Mauch (Ed.), Report 

on the International Seminar on World Trends in Adult Education Research, Hamburg, Germany: UNESCO Institute 

for Education. 

Herrington, A., & Herrington, J. (2006). Authentic learning environments in higher education, Hershey, PA: 

Information Science Publishing. 

Hung, D., & Chen, D. (2001). Situated Cognition, Vygotskian Thought and Learning from the Communities of 

Practice Perspective: Implications for the Design of Web-Based E-Learning. Education Media International, 38 (1), 

3-12. 

Köhler, V. (2006). Co-creators of Scope of action: An exploration of the dynamic relationship between people, IT, 

and work in a nursing context, Licentiate thesis, Luleå University of Technology. 

Lundin, J. (2003). Synchronizing Asynchronous Collaborative Learners. In Huysman, M., Wenger, E. & Wulf, V. 

(Eds.), Communities and Technologies, Dordrecht: Kluwer Academic, 427-433. 

47

Löwgren, J., & Stolterman, E. (2004) Thoughtful Interaction Design: a design perspective on information 

technology, Cambridge, Mass: MIT Press. 

Markus, M. L., Majchrzak, A., & Gasser, L. (2002). A Design Theory for Systems that Support Emergent 

Knowledge Processes. Management Information Systems Quarterly, 26 (3), 179-212. 

Orlikowski, W., & Yates J. (1994). Genre Repertoire: The Structuring of Communicative Practices in Organizations. 

Administrative Science Quarterly, 39, 541-574. 

Roberts, G. F. (1998). The Home Page as Genre: A Narrative Approach. Paper presented at the Thirtieth-First 

Annual Hawaii International Conference on System Sciences, January 6-9, 1998, Hawaii. 

Ryan, T., Field, H. G. R., & Olfman, L. (2002). Homepage Genre Dimensionality. Eight American Conference on 

Information Systems, retrieved October 15, 2007, from http://melody.syr.edu/hci/amcis02_minitrack/CR/Ryan.pdf. 

Saebø, Ø., & Päivärinta, T. (2005). Autopoietic Cybergenres for e-Democracy? Genre Analysis of a Web-Based 

Discussion Board, Paper presented at the 38 th Annual Hawaii International Conference on System Sciences, January 

3-6, 2005, Hawaii. 

Shepherd, M., & Watters, C. (1998). The evolution of Cybergenres. Paper presented at the Thirtieth-First Annual 

Hawaii International Conference on System Sciences, January 6-9, 1998, Hawaii. 

Shepherd, M., Watters, C., & Kennedy, A. (2004). Cybergenre: Automatic Identification of Home Pages on the Web. 

Web Engineering, 3 (3 & 4), 236-251. 

Sims, R., & Jones, D. (2002). Enhancing Instructional Development Processes for E-Learning. Paper presented at 

the E-Learn 2002, 15-19 October, 2002, Montreal, Canada. 

Slough, S., Aoki, J., Hoge, B. & Spears, L. (2004). Development of an E-Learning Framework for Web-based 

Project-Based Learning in Science. Paper presented at E-Learn 2004, 1-5 November, 2004, Washington DC, USA. 

Svensson, L. (2002) Communities of Distance Education, Doctorial dissertation, Department of Informatics, 

Gothenburg University, Sweden. 

Svensson, L., & Östlund, C. (2005). Emergent Design Concepts - An inductive approach to bridging design theory 

and educational practice. WSEAS Transactions on Advances in Engineering Education, 3 (2), 207-217. 

Uskov, V. (2004). Advanced Online Courseware for Student-Centered Learning: The Results of 4-Year NSF CCLI 

Project at Bradley University. Paper presented at E-Learn 2004, 1-5 November, 2004, Washington DC, USA. 

Walls, J. G., Widmeyer, G. R., & El Sawy, O. A. (1992). Building an Information System Design Theory for 

Vigilant EIS. Information Systems Research, 3 (1), 36-59. 

Wenger, E. (1998). Communities of Practice: Learning, Meaning and Identity, New York: Cambridge University 

Press. 

Wills, S., Agostinho, S., Harper, B., Oliver, R., & Hedberg, J. (2002). Developing Reusable Learning Design 

Resources. Paper presented at the E-Learn 2002, 15-19 October, 2002, Montreal, Canada. 

48

Wiberg, M. (2007). Netlearning and Learning through Networks. Educational Technology & Society, 10 (4), 49-61. 

Netlearning and Learning through Networks 

Mikael Wiberg 

Umeå University, Sweden // Tel: +46 90 786 61 15 // Fax: +46 90 786 65 50 // mikael.wiberg@informatik.umu.se 

ABSTRACT 

Traditional non-computerized learning environments are typically founded on an understanding of learning as 

acquiring silence for an effective individual learning process. Recently, it has also been reported that the high 

expectations for the impact of computer-based technology on educational practice have not been realized. This 

paper sets out to challenge both the assumptions made about the requirements for effective learning 

environments by pointing in the direction of social creative learning processes, as well as the technologies for 

effective and creative learning processes by redirecting the focus from what has been labeled “traditional 

computer-based learning environments” towards user-driven learning networks. Thus, this paper proposes the 

concept of netlearning as a general label for the traditional use of computer-based learning environments as 

education tools and then, it suggests the concept of learning through networks as a challenging concept for 

addressing user-driven technologies that support social, collaborative and creative learning processes in, via, or 

outside typical educational settings. The paper is inspired by recent research into the interaction society and the 

Scandinavian tradition in system development that always have highlighted the importance of user-driven 

processes, users as creative social individuals, and a perspective on users as creative contributors to both the 

form, and content of new interaction technologies. The paper ends with a presentation of a participatory design 

project in which children developed their own computer-based tools for editing film and I present this 

technology followed by a discussion on user-driven design of learning technologies in which the technology is 

not just a container for something else, but instead, novel technologies as a tool that directly enable the children 

to do new things, i.e. to collectively learn through their computer-supported social network. 

Keywords 

Device cultures, Interaction, Learning through networks, Mobility. Netlearning, Web 2.0 

Introduction – silence please! 

Think about a typical non-computerized learning environment, e.g. a library, or a student room. One thing obvious 

about these two different settings is that the library design, and the student room layout, as well as the expected 

activities that these two different rooms should support can be characterized by the word “silence”. Traditional noncomputerized 

learning environments are typically founded on an understanding of learning processes as acquiring 

silence for an effective individual learning process. These two aspects of learning processes, i.e. silence and 

individual isolation are two aspects that this paper sets out to challenge from a modern computer-based learning 

perspective, including recent studies into the importance of social interaction and creativity for effective computerbased 

learning processes (Muirhead, 2007). Most recently, it has also been reported that the high expectations for the 

impact of computer-based technology on educational practice have not been realized (e.g. Gifford & Enyedy 1999; 

Muirhead, 2007). According to Beynon (2007) and Norris, et al. (2002) this latest fact has led researchers to place 

greater emphasis on cultural issues. Following this trend, this paper sets out to explicitly explore the concept of 

learning technologies, or netlearning within the socio-technical culture of an emerging interaction society (Wiberg, 

2004). On a more detailed level, this paper sets out to challenge both the assumptions made about the requirements 

for effective learning environments, by pointing in the direction of social creative learning processes, as well as the 

technologies for effective and creative learning processes, by redirecting the focus from what has been labeled 

“traditional computer-based learning environments” towards user-driven learning networks supported by social 

internet based applications. 

Let’s get loud – from isolated & individual learning environments to social networking and 

learning through networks 

While the traditional, individual, and isolated learning environment can function as a calm place for an individual to 

reflect upon e.g. text books, articles or other forms of literature, and while this setting is often romanticized, the 

individual is in fact also restrained by a number of factors. In an isolated learning environment, the single individual 

have no access to a second opinion from another person, no access to a complementary perspective, or external 






49

critique, neither do the single individual have any chances to get complementary literature from anyone which might 

have a different reference library. Given this, there is not much social interactions in this kind of traditional learning 

environment. 

More recently, we can see this focus on isolated learning environments gaining new terrains in forms of e.g. selfstudies 

and “learn yourself x in only three weeks”-book series. Technology wise, we have as a technological field, 

followed a similar trend in our focus on learning technologies that support the individual and his/her learning 

processes. Some of these technologies, e.g. systems for online universities, or distance education do have some 

support for social interaction. However, most of these systems assume a centralized communication model in which 

the learning peers (i.e. the students) mostly communicate with one central peer (i.e. a mentor or advisor). This leads 

in many cases to communication related to the structure rather than the content of an online education and does not 

support spontaneous, creative social learning processes. 

On the other hand, it is today widely acknowledged that social networking is an important aspect of learning 

processes (Castells, 1996), and the following conversation can serve as an illustrative example of how much of what 

we know is in fact just part of threads which includes other persons in our social networks: 

"Do you think me a learned, well-read man?" 

"Certainly," replied Zi-gong, "Aren´t you?" 

"Not at all," said Confucius. 

"I have simply grasped one thread which links up the rest" 

(Recounted in Sima Qian (145-ca. 89 BC), "Confucius," in Hu Shi, The Development of Logical Methods 

in Ancient China, Shanghai: Oriental Book Company, 1922; quoted in Qian 1985:125, in Castells 1996, p. 

1. ) 

This quotation also illustrate that what it is to be knowledgeable can be defined either in terms of how much one 

person have read and learned in isolation, or how knowledgeable a particular person is about different threads to 

grasp in order to gain access to other peers in different social networks. In more general terms, the later 

understanding of what it is to be knowledgeable corresponds to the “learning through networks” concept as proposed 

in this paper. This concept pinpoint the social dimension of learning processes, it pinpoints the social interaction 

setting, and it goes back to a Socratic understanding of knowledge gaining through conversations and argumentations 

with others. 

The concept of “learning through networks” also corresponds well to the current development towards an 

interaction society (Wiberg, 2004) in which IT plays an important role since these new interaction technologies 

enable people to communicate and interact “anytime, anywhere”. In this view, modern interaction technologies 

enable social networks to stay connected even though they might not be collocated, to collaborate over distances, and 

to keep each other updated whenever they need or want to get in touch with each other. In the interaction society, the 

focus is not directed at one-way information flows or information processing. Rather, the focus is on the technologyenabled 

interplay between the habitants of the interaction society, their social networks, their innovations of new 

interaction technologies, and the new channels they invent for their own social learning processes. Today, we can see 

new computer-related behaviors arising as an effect of these interplays including e.g. community building activities, 

the establishment of sharing cultures, and innovative linkages of different interaction technologies (sometimes 

referred to as “mash-ups”). 

In line with this trend, this paper proposes the concept of netlearning as a general label for the traditional use of 

computer-based learning environments as education tools and the concept of learning through networks as a 

challenging concept for addressing user-driven technologies that support creative, social learning processes in, via, or 

outside typical educational settings. 

There is a wide range of literature today that point in the direction of user-driven, innovative and creative processes 

in which modern IT plays an important role as a mediating and enabling platform, and support engine for social 

interaction. For instance, the book “The interaction society” (Wiberg, 2004) illustrates the turn to social interaction 

when it comes to modern IT-use and points out that IT might be better described as an interaction technology, rather 

than just an information technology since much of our IT-use today can be more closely related to supporting social 

50

interaction (e.g. email, skype, ICQ/MSN, etc) rather than information processing (e.g. calculation or transactions) 

from the individuals viewpoint of everyday IT-use. Another example of this kind of literature is the book 

“Connections – New ways of working in the networked organization” (Sproull & Kiesler, 1998) which illustrate the 

complexity of social networks and the highly intertwined role of modern IT as a mediating channel for social 

interaction. 

Finally, in the book “The rise of the Networked Society” Castells (1996) provide an in-depth analysis of how this 

new technology is not only reshaping our everyday IT-use, our social networks and our organizations, but in fact, 

how this technology have large scale implications for our society as a whole. According to his view, our modern 

society is today best described as a network society in which people uses their social and computational networks to 

do business and live their everyday lives. In the next section I take one step back and discuss how this focus on 

interaction relates to the Scandinavian tradition and the emerging interaction society on a general level before 

presenting a more detailed description of interaction as a core concept in learning trough network processes. 

Interaction as a vehicle to support new learning processes 

This paper is inspired by recent research into the interaction society (Wiberg, 2004) as well as it has some of its roots 

in the Scandinavian tradition (e.g. Hirschheim & Klein, 1989; Spinuzzi, 2002) i.e. in Scandinavian system 

development projects that always have highlighted the importance of user-driven processes (see e.g. Ehn, 1988), 

users as creative social individuals (Warr & O´Neill, 2005), and worked for a perspective of users as creative 

contributors to both the form, and content of new interaction technologies (Ehn, et al., 1983). 

Interaction is as such an approach with clear roots in Scandinavia and at the same time it is a modern concept able to 

capture a movement towards an interaction society. In such a society the technology enable new learning cycles and 

new paths for generating new knowledge. As formulated by the International Herald Tribune: 

"We are moving from the information society to the interaction society, and Lunarstorm is leading the 

trend," said Ola Ahlvarsson, chairman of Result, a Stockholm-based technology consulting company. 

"Young people here no longer accept a flow of information from above. They trust what they hear from 

friends on their network." 

This quote describes the story about Lunarstorm, a Swedish Internet service that 90% of high school students in 

Sweden are subscribed to. As this quote illustrates, we are heading towards an emerging interaction society in which 

new technologies enable socially connected, creative, online learning environments. It also illustrates the bottom-up 

approach, or the learning through networks learning cycle. Today, this learning through networks approach can also 

be found elsewhere on the Internet in the form e.g. the Web 2.0 concept, or sometimes also described as usergenerated 

content. 

The basic concept of interaction 

Before going further into the importance of social interaction for creative learning processes we present a close-up 

view of the concept of interaction and discuss how it relates to the concept of interaction technologies, CSCW 

(Computer Supported Cooperative Work) - and CSCL (Computer Supported Collaborative Learning)- technologies. 

The concept of interaction can, according to Dix & Beale (1996), be decomposed into the concepts of 

communication and collaboration. These two dimensions of interaction form two basic levels in the person-personobject 

model as formulated by Ljungberg (1999) based on Dix & Beale (1996) (see figure 1). 

In this model, communication is the exchange of information between people, e.g. Video conferencing. 

Collaboration is when two or more people are operating a common object or artifact (e.g. co-operative authoring 

where the shared document is the common object. In collaboration, operations produce "feedback" to the operator, 

but also "feed through" to co-workers). Support for collaboration is sometimes combined with support for 

communication, e.g., a collaborative authoring system (collaboration) equipped with a chat feature (communication). 

In the context of this model, communication and collaboration can be conceived as subsets of "interaction". As 

51

suggested by Ljungberg (1999b) we can use "CSCW technologies" to frame the technological support for interaction, 

i.e., communication technologies, collaboration technologies, and interaction technologies. 

Figure 1. Definition of the concept of interaction and interaction technologies (From Ljungberg (1999) based on Dix 

& Beale (1996)) 

According to Dix & Beale (1996) the concept of interaction is precisely this combination of person-to-person 

communication, and collaboration around a shared object. In other terms, the concept of interaction can be conceived 

as a unifying concept for communication and collaboration. This social interplay between different people around 

shared objects, and how these people and objects are intertwined in different social structures and contact networks 

are thus the primary focus for studies of the interaction society. 

Today, we can see new such structures arising as an effect of clever design of social network technologies that 

enable social interaction, networking and new ways of learning, i.e. learning through networks. These services 

include interaction technologies like e.g. YouTube, Flickr, Skype, and del.ici.ous, and in the next section we take a 

close-up look at a couple of these services that support this new learning through networks learning cycle. 

Interaction technologies in support of learning through networks 

During the last couple of years we have been able to witness a shift in the development of learning technologies. 

While traditional learning technologies were focused on teacher-centered learning processes (Hiltz & Turoff, 2005; 

Gertzman & Kolodner, 1996) and specific learning objects we can now see how the technology is developed in a 

new direction (Shackelford, 1990; Gifford & Enyedy 1999). Now, the technologies are characterized as being open 

for everybody, general in terms of content, and builds upon a model of user-centered production of content instead of 

centralized, or teacher-centered production of content. Further on, the architecture is flat, and typically peer-to-peer 

to facilitate direct people-to-people interaction, collaboration, and learning. 

These new web-based technologies (including e.g., Skype, Flickr, YouTube, Orcut, LinkedIn, Wikis, blogs, and new 

forms of social network services like e.g. the Four word film review web site www.fwfr.com) enable people to come 

together in new ways to share ideas, opinions, content, humor or ideals and as such, these technologies enable new 

forms of creativity, socialization, and learning. As described by Jenkins and colleagues (2006) these new social 

network services enable and scaffold new “participatory cultures” including; 1) affiliations (memberships, formal 

and informal, in online communities centered around various forms of media, such as Friendster, Facebook, message 

boards,m metagaming, game clans, or MySpace), 2) expressions (producing new creative forms, such as digital 

sampling, skinning and modding, fan videomaking, fan fiction writing, zines, mash-ups), 3) collaborative problemsolving 

(including working together in teams, formal and informal, to complete tasks and develop new knowledge 

(such as through Wikipedia, alternative reality gaming, spoiling), and finally, 4) circulations, i.e. shaping the flow of 

media (such as podcasting and blogging). 

While email and mobile phones are two basic technologies that have enabled people to communicate at a distance, 

we can now see new forms of interaction support being developed. In this section we point at four such digital 

52

services that all demonstrate new ways of learning through networks. More specifically, we will take a closer look at 

the digital services YouTube, Creative Commons, Stumble Upon, and Del.icio.us. 

One thing that these four Internet-based services have in common is that they rely on user-generated content, i.e. the 

company behind the service offers only the digital structure for it and leaves the content to be consumed, and produced by 

anyone interested in contributing to the community forming around the service. If taking a closer look at YouTube 

(www.youtube.com) for instance (see figure 2, left), the service enable people to upload their own short movies to 

the YouTube website, and others can brows the uploaded moveclips and play any clip they find interesting. YouTube 

continuously present “videos being watched right now” by others, as well as a list of featured videos. Besides this 

“social navigation” (Dieberger, et al., 2000) inspired support the site also support standard search of the content on 

the site. On YouTube, everybody can post new move clips, and everybody can rate, comment, or save a clip as a 

personal favorite. Each time a clip is watched it is tracked by a counter so it is always possible to know for how 

many times a certain clip has been played. A clip can also easily be shared by pushing a “share” button in the end of 

each clip to send a URL to anyone over email. 

Another example of a web-based sharing service is Creative Commons (www.creativecommons.org) (see figure 2, 

right). Creative commons has the tag line “Share, reuse, and remix — legally”, and that is the essence of what this 

service is all about. A big issue when it comes to open web-based services for sharing user-generated content has 

been how to deal with copyright issues. On Creative Commons everybody are able to share their content with 

anyone. However, what is different here is respect to YouTube is that e.g. authors, scientists, artists, and educators 

can easily mark their own work with the proper rights they want their content to carry. Anyone who contributes with 

some content to Creative Commons can apply any copyright terms from "All Rights Reserved" to "Some Rights 

Reserved.” Thus, Creative Commons supports a bottom-up, user-driven, and open learning and sharing culture while 

protecting the individual’s rights. 

Figure 2. Screenshots of YouTube (left) and Creative Commons (right) 

In relation to the main argument in this paper, these two services illustrate another important aspect of the learning 

through networks phenomenon. While traditional learning material are directed, produced, packaged, distributed, and 

consumed by learners, new material in these digital networks follow another cycle characterized by user-generated 

content that is continuously spread, re-mixed, re-spread, and so on. As such, this new media follows a new media life 

cycle (Wiberg, 2007). 

The sharing culture is important for social learning processes, for maintaining a community, and for the creation of 

shared points of references. But in these new learning environments it is not only the content that is shared and 

circulated, but also the peers themselves (Hoppe, et al., 2005; Milrad, et al., 2005). Today we can see several similar 

sites growing fast in terms of number of associated and active members including e.g. the Stumbleupon.com site and 

53

the del.icio.us site (see figure 3). Below we take a closer look at these two services from the perspective of new 

learning environments for learning through networks. 

One important aspect for the learning through networks concept to work is to provide the tools for finding new peers 

of content in the network, and to be able to share found peers with other members of the network. Two such 

examples of technical support can be found at the Stumbleupon.com site and the del.icio.us web site. 

Stumbleupon.com (figure 3, left) works actively with the tag line “Discover new sites” and that is also the essence of 

the site. On Stumbleupon.com people can recommend and review sites so that others more easily can get across 

relevant information and find sites related to their own interests. The overall idea with Stumbleupon.com is to serve 

as a public recommendation service. The del.icio.us site (figure 3, right) on the other hand serves a similar, but 

slightly different purpose. Here, the focus is about providing information to make it easier for others to find new 

interesting peers in the network. But, it is now just any peer that one might want to recommend to others. Instead, the 

central unique idea behind the del.icio.us site is that it is assumed that what is interesting to the single individual 

might also be interesting to others. Thus, on the del.icio.us site anyone can share their personal favorites (similar to 

“my bookmarks” in a browser) to anyone else. As such, the del.icio.us site builds upon an idea of finding new 

interesting material in the network through the glancing at other points of references in the network. 

Figure 3. Screenshots from the StumbleUpon site (http://www.stumbleupon.com/), (left), and the del.icio.us site 

(http://del.icio.us/), (right) 

Del.icio.us can thus be thought of as a new way of sharing your points of reference, which is indeed an important 

aspect of any body of knowledge. While Albert Einstein ones said, “The secret to creativity is knowing how to hide 

your sources” we can now start to see the creative power behind mass interaction around shared sources. The 

del.icio.us site is only one example. Another example is Wikipedia, the online, user-generated, and user-maintained 

dictionary, which contains almost the same amount of information as the Webster´s dictionary (although its accuracy 

has been frequently debated during the last two years). 

With all these new services in place for uploading and sharing new content across the Internet it also becomes 

obvious that there are not only needs for tools to find new peers, but also to keep track of changes at already known 

peers of interest (Hoppe, et al., 2005; Jones, et al., 2005, Chen, et al., 2007). Today, we can therefore see a growing 

interest in e.g. RSS- and similar technologies to enable people to automate the checking for new content on remote 

peers (Miao, et al., 2005; Glotzbach, et al., 2007), and mash-up technologies (Ankolekar, et al., 2007), (including e.g. 

yahoo pipes) to enable people to more easily combine different sources of information, and also form new services 

(to again support the new learning vehicle of uploading, sharing, and circulating interesting content across the 

networks. 

54

Taking one step back and two steps forward – towards new physical (and social) learning 

environments 

If taking one step back for a moment to have a look at the direction in which the development of many traditional 

learning environments are heading today, we can notice that there is a similar trend going on around traditional, 

physical learning environments as well. In the classical library, silence and individual search for literature was key 

factors for any library design. Now however, we have a new set of design requirements for these classical learning 

environments. 

For instance, if reviewing the redesign of the city library in Umeå (see figure 4), a city in the north of Sweden, we 

can notice a couple of fundamental changes. In the old library, bookshelves were arranged as to create narrow 

corridors all across the library. Now, there is the same amount of books, but 1/3 of the books have been placed on a 

newly built upper floor. A stair with glas sealing enable the library visitors at the ground floor to see other people 

moving up and down the stairs. And, it enables people using the stairs to get a good overview of the library area 

below. Further on, the upper floor only covers 1/3 of the ground floor. This enable people at the upper floor and 

ground floor to see each other and communicate if they which to do that. Finally, a new sofa was installed around 

one of the scaffolding piles in the library. An important aspect of this sofa is its central place in the new open area, 

and another important aspect is that the seats are faced outwards. This might not be ideal for a sofa conversation, but 

it fulfills the idea of an open library in which people notice and acknowledge each other. According to the architects 

behind this redesign, a guiding principle was “social awareness”, a not so unfamiliar word in the area of information 

technologies these days. 

Figure 4. Illustration from the redesign of the city library in Umeå as an example of a new approach to traditional 

learning environments 

What we can learn from this example is that the “learning through networks” trend is more far reaching than we can 

grasp if looking solely at digital and specific learning environments. Instead, we need to redirect our attention and 

acknowledge that this trend moves across the digital and the physical landscape, and across new, as well as 

traditional, old-style, learning environments. While the traditional library was about silence, and individual browsing 

of literature, the new library might be better labeled as”the social interaction library”. While this might be to 

extreme, this example still illustrates the movement towards more open and more socially encompassed learning 

environments. 

55

Device cultures 

It is not only the physical learning environments that are affected, effected, and changed as an effect of this 

movement towards socially encompassed learning settings. Another change is related to where the learning 

processes, in terms of knowledge sharing, interaction, and collaboration take place. In the knowledge society, 

sometimes referred to as an “interaction society” (Wiberg, 2004) people bring along a myriad of interaction tools and 

digital devices (typically mobile phones, laptop computers, or different handheld devices) to enable them selves to 

interact with anyone, at anytime, and anywhere. These devices and the appropriation of these devices are interesting 

in several different ways. The way in which people customize their devices to make them fit their needs and 

everyday behaviors is very individual, and it tells a lot about the individual who customize the devices. At the same 

time, the adoption and customization of devices is not only an individual processes, but also a highly social process. 

In the customization of my own devices, I do at the same time do this in relation to others. For instance if my friends 

are running Skype, or similar VoIP services on their computers it is likely that I will also install and set up my own 

Skype account on my device. The customization of digital devices is therefore both an issue for the individual, as 

well as it is a highly social process. In our view, this is also an example of a delicate symbiotic interplay concerning 

the acknowledging of the peers in the social networks, and simultaneously about the acknowledging of oneself and 

ones relation to the peers that these networks consist of. In the end, we are just individual peers in these new social 

networks, but this time around we have the technology available for reaching out to interact with others who might 

not be, in geographic terms, close to us. 

The device culture, characterized by people wearing, carrying, using and constantly configuring digital devices are 

now possible to observe allover our modern society. People see these devices as their ontological security (Lowry & 

Moskos, 2005), think that they cannot manage without them (Fortunati, 2001) and carries them with them to ensure 

constant accessibility to their online social networks (e.g. Wiberg & Whittaker, 2005; Sadler, et al, 2006). From a 

learning through networks perspective, and in line with Lankshear & Knobel (2006) it is a way to ensure their 

belonging to their social networks as an important point of reference for their learning, and collaborative processes. 

The device culture is also characterized by a few additional factors. The device cultures are about the use and 

appropriation of digital devices, but also about a few additional aspects in relation to the learning through networks 

phenomenon. First of all, we have the “I, I, I trend”, i.e. we can see the acknowledging of the individual in services 

like iTunes and iGoogle. But at the same time, we can also see digital services that build upon the strength of 

individual contributions to the public, which is about our social networks, shared digital content, and mass 

interaction (Keen, 2007) including e.g. the most recently, and highly popular internet service Facebook 

(www.facebook.com). Further on, we should not underestimate the importance of fun as a mean to support creative 

learning processes. This has been acknowledged previously in the literature (e.g. by Neal, et al., 2004 and 

MacFarlane, 2005) and specific terms, like “edutainment” (e.g. Rasmussen, 1994 and Rapeepisarn, et al., 2006), has 

been formulized to highlight the importance of fun for good learning processes. Finally, we should not forget about 

the coming generation, i.e. the children as an important component of today’s device cultures. These young people 

are often early adopters of new technology, creators of innovative products and services and adaptive to learn, and 

spread new things (Jenkins, et al., 2006). Without making it into a cliché, the young people of today are the nodes in 

tomorrow’s networks. 

To summarize this trend, which today is typically described or labeled as a movement towards “Web 2.0” we can 

now make the following distinctions between “Web 1.0” and “Web 2.0”: In “Web 1.0” most web sites were quite 

static in terms of frequency of information updates, designed to be centrally (and seldom) administrated, corporatedriven, 

and seen as a showroom for internet visitors. On the other hand, for “Web 2.0” services the focus is on userdriven/invented 

Internet projects, user-generated content, socially exchanged digital media via these new digital 

services, and characterized by a high frequency of updates, which typically include various forms of digital media 

rather than text based corporate information. 

Supporting children to learn film editing through participatory design and social 

collaboration 

While it might be fine to just theorize about the important aspects of devices cultures and the development towards 

Web 2.0 in relation to the phenomenon of learning though networks it might be equally important to illustrate these 

56

ideas in a concrete project. In this section I therefore set out to take the aspects of learning through networks in the 

modern device cultures to an applied setting and present a participatory design project in which children developed 

their own computer-based tools for editing films and movie clips. On a general level, our project is related to e.g. the 

work of Hiroaki Ogata, specially their project with RFID Tags for language learning (http://www-yano.is.tokushimau.ac.jp/ogata/projects.html), 

the research conducted by Masanori Sugimoto on tangible learning environments (e.g. 

Sugimoto, et al., 2003), the research conducted by Kurti, et al. (2007) in the AMULETS (Advanced Mobile and 

Ubiquitous Learning Environments for Teachers and Students) project, and the Savannah project at FutureLab in the 

UK (http://www.futurelab.org.uk/resources/documents /project_reports/ Savannah_research_report.pdf). 

In this section, I present our project and the technology we have developed followed by a discussion on user-driven 

design of learning technologies in which the technology is not just a container for some other learning objects, but 

instead, a novel technology that in itself serves as a tool that directly enable children to do new things, i.e. to 

collectively learn through their computer-supported social network, i.e. a digital support for learning through 

networks. 

In 2005, we initiated a participatory design project together with 12-year old children as part of the EU funded 

Meetings/vITal research project. During 6 mounts, a group of 13 children from a local school participated in a 

creative process of identifying needs for novel collaboration technologies. Their participation included the needs 

identification process, the system requirement analysis, and the design of a concrete system to support their vision of 

a novel collaboration technology. In this project, the children developed a huge number of paper mock-ups, interface 

sketches, and low-fi prototypes to illustrate, and communicate their visions concerning novel technical support for 

editing film (see figure 5). Their main vision included, amongst other ideas, a circular, table size, area for tangible 

editing of movie clips by just arranging physical tags (RFID-tags) in the order they should be played in the finalized 

video, and the whole project is carefully described in Vaucelle, et al., (2005). 

Figure 5. Two illustrations from the creative process of identifying the needs and visions for a future computer-based 

novel tool for editing movie clips. To the left, a picture from one of the sessions with the children, and to the right, an 

example of an interface sketch drawn during this project 

From a “learning through networks” point of view, however, there are some interesting aspects to acknowledge here. 

As formulated above, we took those principles as a guide and involved children as creative and active participants in 

this project. We also allowed them to freely play around with paper and pencil materials as to make the process fun. 

We stimulated them to work together rather than in isolation, and we wanted them to develop a tool that could work 

for them as an individual as well as being a good support for collectively editing film. 

To us, it turned out that these criteria’s worked very well, and in only 6 mounts, the children went from vague initial 

ideas to an implemented system (although they had some help in the end with the programming of the actual system 

and the creation of the table, the tags, and the special cameras developed in the project. Figure 6 illustrate the final 

version of the system which contains 1) a circular editing table that the children can use to arrange their tags (RFIDtags) 

on in any order they want them to appear in the final cut of the movie, 2) a special camera developed that was 

57

uilt using a camera-equipped PDA running windows mobile which we also equipped with a WLAN-card and an 

RFID tag reader (as to enable association of video clips with RFID-tags, and as a seamless way to transfer the video 

clips taken to the editing table (over WLAN). As illustrated in figure 6 (right) the children could then just take an 

RFID-tag. Place it in the holder on the camera (see the little light blue circle), shot a sequence, and then just take the 

RFID-tag from the camera and place it in a proper place on the editing table. 

Figure 6. The editing table (left), the RFID/WLAN-camera (center), and a couple of the children in the project 

playing around with the cameras and the editing table (right) 

So, while this project on the one hand illustrates the design of a novel technology for film editing in terms of a 

robust, tangible interface for collaborate film editing, it does at the same time illustrate something else of importance 

from a learning through networks perspective, i.e. the power of social, user-driven design of learning technologies in 

which the technology is not just a container for something else, but instead, a tool that directly enable the children to 

do new things, i.e. to collectively learn through their computer-supported social network. For the children that 

participated in this project, the most important lesson might not be how to design this specific tool, but rather the 

discovery of the power of playing around together, playing around with digital technology, and participate in a 

design process of new technologies Vaucelle, et al (2005), and how such a process can stimulate creativity, and thus 

stimulate new ways of learning. 

Conclusions – A final note on netlearning vs. learning through networks 

In this paper we have elaborated on the notion of netlearning as a labeling concept for learning technologies, vs. 

learning thought networks as a label for the new ways in which people use the Internet and wireless mobile networks 

to communicate, collaborate, participate and generate new knowledge around their own special topics. As such, this 

paper have challenged both the assumptions made about the requirements for effective learning environments by 

pointing in the direction of social, creative learning processes, as well as the technologies for effective and creative 

learning processes by redirecting the focus from what has been labeled “traditional computer-based learning 

environments” towards user-driven learning networks. 

Further on, we have in this paper proposed the concept of netlearning as a general label for the traditional use of 

computer-based learning environments as education tools and then we have proposed the concept of learning 

through networks as a challenging concept for addressing user-driven technologies that support creative, social 

learning processes in, via, or outside typical educational settings. The general outline for this paper has been inspired 

by recent research into the interaction society and the Scandinavian tradition in system development that always have 

highlighted the importance of user-driven processes, users as creative social individuals, and a perspective on users 

as creative contributors to both the form, and content of new interaction technologies. 

58

Finally, we have in this paper presented a participatory design project in which children developed their own 

computer-based tools for editing film and we have presented this technology followed by a discussion on user-driven 

design of learning technologies in which the technology is not just a container for something else, but instead, a 

novel technology that in itself serves as a tool that directly enable children to do new things, i.e. to collectively learn 

through their computer-supported social network, i.e. a digital support for learning through networks. 

While this paper started out with a quick step away from the traditional learning environment and the idea of silence 

as an important factor for effective individual learning processes I think that the biggest challenge for further 

research in this field is about generating new knowledge about the other side of the individual “silence coin”., i.e. 

research efforts directed towards understanding and generating new theories and concepts that can explain and shed 

new light upon the processes related to collaborative and creative concentration as a result of learning through, and 

playing with, new forms of digital networks. 

References 

Ankolekar, A., Krötzsch, M., Tran, T., & Vrandecic, D. (2007). Semantic web and web 2.0: The two cultures: 

mashing up web 2.0 and the semantic web. Paper presented at the 16th International Conference on World Wide 

Web, May 8-12, 2007, Banff, Canada. 

Castells, M. (1996). The Rise of the Network Society, Oxford: Blackwell. 

Chen, Y., Fabbrizio, G., Gibbon, D., Jora, S., Renger, B., & Wei, B. (2007). Geotracker: geospatial and temporal 

RSS navigation. Paper presented at the 16th International Conference on World Wide Web, May 8-12, 2007, Banff, 

Canada. 

Dieberger, A., Dourish, P., Höök, K., Resnick, P., & Wexelblat, A. (2000). Social navigation: techniques for building 

more usable systems. Interactions, 7 (6), 36-45. 

Ehn, P. (1988). Work-Oriented Design of Computer Artifacts, Stockholm: Arbetslivscentrum. 

Ehn, P., Kyng, M., & Sundblad, Y. (1983). The UTOPIA Project: On training, technology, and products viewed 

from the quality of work perspective. In U. Briefs, C. Ciborra, and L. Schneider, (Eds), Systems Design For, With 

and By the Users, Amsterdam: North-Holland, 439-449. 

Fortunati, L. (2001). The Mobile Phone: An Identity on the Move. Personal and Ubiquitous Computing, 5 (2), 85- 

98. 

Gifford, B., & Enyedy, N. (1999). Activity centered design: towards a theoretical framework for CSCL. Paper 

presented at the 1999 conference on Computer support for collaborative learning, CSCL '99, 11-12 December 1999, 

Stanford, USA. 

Glotzbach, R., Mohler, J., & Radwan, J. (2007). RSS as a course information delivery method. Paper presented at 

the International Conference on Computer Graphics and Interactive Techniques, 7-9 August 2007, San Diego, 

California, USA. 

Hirschheim, R., & Klein, H. (1989). Four paradigms of information systems development, Communications of the 

ACM, 32 (10), 1199-1216. 

Hoppe, H, Pinkwart, N, Oelinger, M, Zeini, S, Verdejo, F, Barros, B, & Mayorga, J.I. (2005). Building Bridges 

within Learning Communities through Ontologies and "Thematic Objects. Paper presented at the CSCL 2005 

Conference, 30 May - 4 June 2005, Taipei, Taiwan. 

Jenkins, H, Clinton, K., Purushotma, R., Robison, A., & Weigel, M. (2006). Confronting the Challenges of 

Participatory Culture: Media Education for the 21st Century, retrieved October 15, 2007, from 

http://www.projectnml.org/files/working/NMLWhitePaper.pdf. 

59

Jones, C., Dirckinck-Holmfeld, L., & Lindtröm, B. (2005). CSCL - the next ten years: a view from Europe. Paper 

presented at the CSCL 2005 Conference, 30 May - 4 June 2005, Taipei, Taiwan. 

Keen, A. (2007). The Cult of the Amateur: How Today's Internet Is Killing Our Culture, Currency, USA: Trade 

Cloth. 

Kurti, A., Milrad, M., & Spikol, D. (2007). Designing Innovative Learning Activities Using Ubiquitous Computing. 

Paper presented at the ICALT 2007 Conference, July 18-20, 2007, Niigata, Japan. 

Lankshear, C., & Knobel, M. (2006). New Literacies: Everyday Practices and Classroom Learning, Berkshire, UK, 


Lowry, D., & Moskos, M. (2005). Hanging on the Mobile Phone: Experiencing work and spatial flexibility. Paper 

presented at the 4 th Critical Management Studies Conference, 4-6 July 2005, Cambridge, UK. 

MacFarlane, S., Sim, G., & Horton, M. (2005). Assessing usability and fun in educational software. Paper presented 

at the 2005 Conference on Interaction Design and Children, June 8-10, 2005, Boulder, CO, USA. 

Miao, Y., Hoeksema, K., Hoppe, H., & Harrer, A. (2005). CSCL scripts: modelling features and potential use. Paper 

presented at the CSCL 2005 Conference, 30 May - 4 June 2005, Taipei, Taiwan. 

Milrad, M., Björn, M., & Jackson, M. H. (2005). Designing networked learning environments to support intercultural 

communication and collaboration in science learning. International Journal of Web Based Communities, 1 (3), 308- 

319. 

Neal, L., Miller, D., & Perez, R. (2004) Online learning and fun. eLearn, retrieved October 15, 2007, from 

http://www.elearnmag.org/subpage.cfm?section=articles&article=4-1. 

Norris, C., Soloway, E., & Sullivan, T. (2002). Examining 25 Years of Technology in Education, Communications of 

the ACM, 45 (8), 15-18. 

Rapeepisarn, K., Wai Wong, K., Che Fung, C., & Depickere, A. (2006). Similarities and differences between "learn 

through play" and "edutainment". Paper presented at the 3 rd Australasian conference on Interactive entertainment, 

December 4-6, 2006, Perth, Australia. 

Rasmussen, M. (1994). Interactive edutainment on the Internet. ACM SIGGRAPH Computer Graphics, 28 (2), 139. 

Russell, L., & Shackelford, R. (1990). Educational computing: myths versus methods - why computers haven't 

helped and what we can do about it. Paper presented at the Conference on Computers and the Quality of Life, 

September 13-16, 1990, Washington DC, USA. 

Sadler, K., Robertson, T., & Kan, M. (2006). "It's always there, it's always on": Australian freelancer's management 

of availability using mobile technologies. Paper presented at the 8 th Conference on Human-Computer Interaction 

with Mobile Devices and Services, September 12-15, 2006, Espoo, Finland. 

Spinuzzi, C. (2002). A Scandinavian challenge, a US response: methodological assumptions in Scandinavian and US 

prototyping approaches. Paper presented at the 20th annual international conference on Computer documentation, 

October 20-23, 2002, Toronto, Canada. 

Sproull, L., & Kiesler, S. (1998). Connections - New ways of working in the networked organization, Cambridge, 

The MIT Press. 

Sugimoto, M., Kusunoki,F, Inagaki, S, Takatoki, K, & Yoshikawa, A. (2003). Design of a System and a Curriculum 

to Support Group Learning for School Children. Paper presented at the CSCL 2003 Conference, June 14-18, 2003, 

Bergen, Norway. 

60

Vaucelle, C., Africano, D., Davenport, G., Wiberg, M., & Fjellstrom, O. (2005). Moving Pictures: Looking Out / 

Looking In. Paper presented at the SIGGRAPH 2005 Conference, 31 July – 4 August 2005, Los Angeles, USA. 

Warr, A., & O'Neill, E. (2005). Understanding design as a social creative process. Paper presented at the 5 th 

Conference on Creativity & Cognition, 12-15 April 2005, London, UK. 

Wiberg, M. (2004). The Interaction Society - Practice, Theories, and Supportive Technologies, Hershey, PA: Idea 

group. 

Wiberg, M., & Whittaker, S. (2005). Managing availability: Supporting lightweight negotiations to handle 

interruptions. ACM Transactions on Computer-Human Interaction, 12 (4), 356-387. 

Wiberg, M. (2007). Midgets - Towards truly liquid media. Paper presented at the CMID ´07 - The 1st International 

Conference on Cross-Media Interaction Design, March 22-25, 2007, Hemavan, Sweden. 

61

Milrad, M., & Spikol, D. (2007). Anytime, Anywhere Learning Supported by Smart Phones: Experiences and Results from the 

MUSIS Project. Educational Technology & Society, 10 (4), 62-70. 

Anytime, Anywhere Learning Supported by Smart Phones: Experiences and 

Results from the MUSIS Project 

Marcelo Milrad and Daniel Spikol 

Center for Learning and Knowledge Technologies (CeLeKT), Växjö University, Sweden 

marcelo.milrad@msi.vxu.se // daniel.spikol@msi.vxu.se 

ABSTRACT 

In this paper we report the results of our on-going activities regarding the use of smart phones and mobile 

services in university classrooms. The purpose of these trials was to explore and identify which content and 

services could be delivered to the smart phones in order to support learning and communication in the context of 

university studies. The activities were conducted within the MUSIS (Multicasting Services and Information in 

Sweden) project where more than 60 students from different courses at Växjö University (VXU) and Blekinge 

Institute of Technology (BTH) participated during the course of their studies. Generally, the services integrated 

transparently into students’ previous experience with mobile phones. Students generally perceived the services 

as useful to learning; interestingly, attitudes were more positive if the instructor adapted pedagogical style and 

instructional material to take advantage of the distinctive capabilities of multicasting. To illustrate, we describe a 

number of educational mobile services we have designed and implemented at VXU and BTH. We conclude with 

a discussion and recommendations for increasing the potential for successful implementation of multicasting 

mobile services in higher education, including the importance of usability, institutional support, and tailored 

educational content. 

Keywords 

Ubiquitous Learning, Educational Mobile Services, Smart Phones. 

Introduction 

In the past decade, the Internet has spawned many innovations and services that stem from its interactive character. 

The emergence of ubiquitous and inexpensive microprocessors and wireless networks has lead to the wide 

deployment of mobile devices that allow us to access and to handle information almost anytime and anywhere 

(Roussos et al., 2005). Diverse multimedia applications have flourished with recent advances in hardware and 

network technology with the proliferation of inexpensive video-capture devices and widespread adoption of the 

worldwide web via these mobile devices. All these forms of interactive multimedia and communication offer new 

possibilities for supporting innovative ways of learning, collaborating and communicating (Milrad, 2003; Thornton 

& Houser, 2004). These technologies and new forms of mobile communication and collaboration have been widely 

adopted by young people and integrated into their everyday lives. Clear indications of this trend can be found in sites 

such as www.youtube,com, www.flickr.com, and www.facebook.com. However, this transformation does not live up 

to the promises and expectations when it comes to the use of mobile technologies at schools and universities (Norris 

et al., 2002; Tatar et al., 2003). 

Lankshear and Knoble (2006) claim that formal education ignores some of these trends and argue that mobile and 

wireless technologies and new media might be integrated into current school educational activities, as they are 

transforming and defining new literacies in teaching and learning. Thus, there are a number of challenging questions 

that deserve further exploration. What are the implications of using mobile computing and wireless communication 

for supporting learning and teaching? What new scenarios and applications will emerge? In order to understand the 

possible impact of using smart phones for facilitating learning and teaching, we will proceed by presenting the 

results of one of our on-going projects, MUSIS (MUlticasting Services and Information in Sweden). 

This paper presents the results of two pilots studies conducted within the framework of the MUSIS project between 

the periods of 2005 and 2007. By presenting these two periods of the trials, we hope to gain new insights regarding 

how attitudes and expectations towards using mobile phones in educational settings may have changed over the last 

two years. The next section describes the MUSIS project and the technical infrastructure. The method section 

describes the implementation of the trials and the data collection techniques we have used. The results and discussion 

section describes the outcome of our trials and explain how students experienced the mobile services. Issues and 






62

problems are discussed with regard to the technology and its use. Overall conclusions are provided in the final 

section of the paper. 

A brief overview of the MUSIS project 

The main objectives of the MUSIS projects are to explore, identify and develop a number of innovative multicast 

mobile services to support learning with multimedia information to be distributed over wireless networks using 

multicasting solutions at university. The project has had two pilot phases, the first one during 2005 and the second in 

2007. MUSIS (http://www.musis.se) has brought together different partners. The key partners have been TeliaSonera 

(TS), Sweden's largest telecom operator, the City of Stockholm, Växjö University (VXU), and Bamboo 

MediaCasting, a company pioneering in the field of cellular multicasting. Also, Luleå Technology University (LTU), 

the Royal Institute of Technology (KTH) in Stockholm and the Blekinge Institute of Technology (BTH) have been 

actively involved in the project. 

Multicasting mobile services developed in the MUSIS projects are organized as a range of content channels to which 

users can subscribe. Each user can build a personal portfolio of channels that interest them. Multimedia content is 

sent, according to a predefined time schedule, to subscribers over the GPRS (General Packet Radio Service) network 

using wireless multicast technology (Varshney, 2002). It is also possible to program the MUSIS system in order to 

send content to the phones based on discrete events. The content sent to the phone is downloaded in the background 

and stored on the phone’s memory card. Once the content has arrived, the phone beeps announcing a new message 

has been received, similar to standard message services. Users can then interact with the MUSIS client installed in 

the smart phone in order to view and save the content. This approach differs from the latest type mobile services 

offered by the telecom industries, which are using streaming technology. The digital content used in these trials 

included TV news, music, entertainment videos, general information related to student’s activities, such as lecture 

notes (including video and audio), and specific information related to the different courses. 

During the second phase of the project, we introduced additional content tools for the users allowing them to 

multicast video, audio, images, and text directly from the handset. We also expanded the web interface that 

controlled the subscriptions to include the ability to upload and convert content for multicast delivery. This 

fundamental change in this trial focused on shifting the traditional broadcast model we used in phase 1 of the project 

from a one to many model, to a many-to-many model, thus providing students with the ability to explore how these 

concepts could be used in an educational environment. The content in this phase was created by the students and the 

instructors and it included text, calendar events, photographs, and video. All these materials were sent between the 

students´ groups and back and forth between the instructors and the students. 

Technical aspects 

A complex technical infrastructure has been developed in order to deliver the different mobile services to the 

students. This task requires complex software solutions in order to connect and combine the content coming from 

different content providers. Figure 1 illustrates the generic technical architecture and the different hardware and 

software components used in the project. 

Bamboo’s equipment provides the multicasting feature in the GPRS network. The content management system 

(CMS) located at TS is responsible for scheduling the content transmissions. The MUSIS CCS (Collect, Convert and 

Send) developed and implemented at VXU is responsible for collecting, organizing, and converting the different 

digital material coming from all content providers (including educational material produced by the teachers) as 

described above. The MUSIS CCS system provides tools to manipulate content automatically and transmit it to 

Bamboo's router for distributing to the users. The CCS can get the content from the content's resources based on predefined 

rules, convert it to formats that are supported by the mobile handset and transmit it to Bamboo's server. 

These activities can be done automatically without human intervention. 

The following illustration (see figure 2) describes the generic architecture of the MUSIS CCS system. As seen in the 

illustration below, the system is based on several different inputs and outputs. The system, which has been 

implemented using Java related technologies, is scaleable and it consists of modular, reusable and easily expandable 

63

components to be able to deal with new types of content. This includes all features, i.e. the collecting, converting and 

the sending mechanisms. The system is programmed in Java using JSP and Java Beans on a Linux platform. It also 

uses open-source tools and applications. 

Figure 1. MUSIS generic architecture 

Figure 2. Generic illustration of the CCS system (Collect, Convert and Send) 

64

In the next section we concentrate on those activities carried out at Växjö University and Blekinge Institute of 

Technology. We present the results of a couple of pilot studies conducted over a two years period, focusing 

specifically on the question of whether students would find a mobile phone useful for supporting their learning, and 

in particular whether multicasting mobile services would be suitable for supporting learning and other activities 

related to their academic life. These studies aimed to look at the patterns of use of the various mobile services and 

the impact on students’ learning habits. We were also interested in determining what type of functionality is required 

for educational mobile services to be considered useful. Our results lead us to advocate a comprehensive approach 

regarding the introduction of smart phones and mobile services in university classes that considers not only technical 

features but also the individual, social and organizational aspects of technology adoption. 

Method 

Participants 

For the first set of trials, we solicited volunteers from students enrolled in two courses offered at VXU during the 

spring term of 2005. One course was offered at the School of Humanities, and the other at the School of Mathematics 

and Systems Engineering. After a short presentation delivered by members of the research team at the beginning of 

the term, students from these two courses volunteered to participate in the pilot. Twenty-two students from the 

course in the School of Humanities and nineteen from the School of Mathematics and Systems Engineering 

volunteered. Each volunteer was given a ”smart phone” for the duration of the school term (3 months). Although the 

number of smart phones available limited the number of participants, we were able to provide phones to all students 

who wanted to volunteer as participants. Each student signed a contract of use that specified their obligation to 

participate in the project in return for free use of the phone and a small amount of money they could use to make 

phone calls. The project also provided continuously available online and face-to-face support. The project began with 

a workshop session to familiarize the students with the smart phone and the software. Participants ranged from 19 to 

40 years of age, with a mean age of 26. Nineteen were female and twenty-two male. All 41 students already owned 

at least one mobile phone at the start of project. With regard to the issue of how much they spent on their own phone 

services before joining the project, on average a student in this group paid 28 USD a month. Twenty per cent of the 

41 students participating in this study spent more than 45 USD a month. 

For the second trial, we worked with BTH students during the spring term of 2007 in a special project course in the 

Literature Culture and Digital Media in the Humanities program (END011). Twenty-one students and two instructors 

participated in the trial over a five weeks period. Participants ranged from 20 to 26 years of age, with a mean age of 

24. Eleven were female and ten male. The students organized themselves into five groups consisting of 4 persons 

each. The expected outcome of the course for the students was to produce 5 pilot mobile applications that will help 

tourists to explore the history of the local city in novel ways using mobile phones and interactive storytelling 

techniques. The students in this trial all owned at least 1 mobile phone and spent a similar amount in phone costs 

compared to the 2005 trial. 

Equipment and Services 

The participants of the studies were each equipped with smart phones. For the first trail the students were supplied 

with NOKIA 6630 phones and for the second trial NOKIA N70 phones were used. Both phone models run on 

Symbian based operating systems and have mobile internet browsers, cameras with digital zoom, video, still, and 

audio recording, and RealOne’s player for playback and streaming of 3GPP-compatible and RealMedia video clips. 

Additional applications include a personal information management (PIM), a calendar, and a contacts database. 

Users could synchronize contacts and calendar stored on the phone with data stored on a personal computer. Since 

the Symbian operating system is open we were able to develop a Python application that enables mobile multicasting 

from the handset. This particular feature was implemented during the second trial. 

Technical development of MUSIS services took place concurrently in both studies, enabling refinements during the 

project cycles. For the first pilot phase that took place during the period March 1st- April 30th 2005, all participants 

accessed the same set of channels, receiving approximately 5 to 7 MUSIS messages (push technology) daily. One of 

these channels carried educational content related to their VXU course. Subscription to the educational channel was 

65

compulsory throughout the project. However, beginning May 1, 2005 users were able to subscribe to up to 30 

channels of their choice using a Web interface (both available via a PC or a mobile phone) specially developed for 

this project. During the second phase that took place during April 1 to May 24, 2007 all participants and two 

instructors accessed two public channels and then each of the five groups had a group channel for inter-group 

communication. In this paper, we focus specifically on our experience and results with the different educational 

channels only. 

Figure 3. The MUSIS client interface (left) and the mobile multicast client interface (right). 

Implementation of phase 1 

For the first trial, educational materials delivered for this project include small ”micro lectures” in video format, 

voice based course information and assignments, and specific information related to the logistics (calendar 

information, cancellation of lectures and so on) of the different courses. In the case of the ”micro lectures” the audio 

based and text information, the contents were developed for (and sometimes tailored to) the phone by the course 

instructor. In order to send this material to the phones, the teacher used a special web interface we designed for this 

purpose. We also developed a number of solutions that allow internet-based educational resources used in the course 

to be sent automatically to the phones. Instructors were also given a smart phone of the same type given to the 

students. 

FirstClass (FC) is a communication platform used at Växjö University mainly for distance education but also for 

campus based courses. There are two ways of accessing the FirstClass application. Students can use the FC client 

software or a web-based client directly from any browser. In the current version implemented at VXU, the only way 

to deliver FC content to mobile phones is by purchasing a very expensive SMS module. We developed an application 

using java and XML that it is used to convert the instructor’ contributions in the FC forum to an RSS (Real Simple 

Syndication, an XML format for syndicating web content) feed that it is then multicast to the phones. The java 

application was running in the background of the FC forum, so the instructor’s contributions to the forum were 

automatically transformed to a format suitable for the phone. The content from the FC forum arrives to the phones as 

a file in HTML format that can be viewed with the phone’s Internet browser. 

Implementation of phase 2 

In the second trial, the MUSIS system was used to foster collaboration and communication between the instructors 

and the students. Figure 4 below illustrates how communication and collaboration between the teachers and the 

student groups was envisioned. Instructors and students could multicast to group members, the entire class, specific 

groups and instructors using the mobile application we developed for the handset, as well as the development of the 

web-based interface to achieve the same goals. The system and the smart phones supported group and class 

interaction. This allowed the students to schedule group work and giving the instructors an additional way to give 

feedback using video, audio, and text messages to the class and the students´groups. This also provided ways to 

coordinate and organize the class work with calendar and text based messages sent to the phones. 

66

Figure 4. Communication and collaboration between instructors and student groups 

In this second phase, we modified the multicast model to allow any user at anytime and from anywhere to multicast 

content from the smart phone. This fact opened up different ways to use the technology not only in educational 

contexts but also to support students´ daily activities. The key application that enabled this feature was a python 

application we developed enabling digital content generated with the smart phones (including video, still 

photographs, and audio) to be uploaded to the system and then multicast as an event to a particular channel. In 

addition, the application also provided plain text and calendar events that could be multicast. 

Data collection across both projects 

Given the exploratory nature of these studies, we used multiple methods to collect data for both phases of the MUSIS 

project. This allowed us to scan the patterns of uses and attitudes that could be investigated more specifically in 

future studies. For the first phase participants completed web based questionnaires in weeks 1, 5, and 10 of the 

project. The first survey included items that measured personal attitudes toward mobility, attitudes toward media 

formats, and how much different media formats were used. The second and third surveys included items regarding 

perceived effect of the phones on learning, preference for different media formats, preference for channels, and 

perceptions of telephone functionality and usability. Additionally, members of the research team facilitated four 

focus group interviews with 15 participants, which were videotaped. 

The focus group ranged in size from 3 to 6 participants. The interview covered issues regarding the participants’ 

perception of the project in connection to the services, the functionality and usefulness. Additionally, the participants 

were asked to suggest and discuss additional educational mobile services that could be developed. Finally, a 90 

minutes workshop with the students was held at the end of the term, which was videotaped. The purpose of these 

sessions was to carry out an open discussion with most of the students in order get an overall view of how the 

students experienced the project. In the second phase of the project we continued using multiple methods to for data 

collection. Focus groups sessions with each of the groups were videotaped, covering perception and future uses of 

the technology. Additionally, the communication data was collected providing some additional insight on how 

collaboration and communication in each group and how the mobile application supported this. The main objectives 

of all these activities were to assess the usefulness and quality of the services, to identify problems experienced by 

the students and to explore how future MUSIS services could look like. 

Results and discussion 

General use and attitudes 

The majority of the students participating in the two MUSIS pilots had mobile phones of their own before they 

joined the project. Therefore, delivering content directly to the smart phones is transparent for them, integrating not 

67

only with their existing day-today practices, but also with their views of mobility and accessibility as central to their 

life-style. It is important to note that even if their personal handsets supported a variety of features such as e-mailing, 

surfing the Internet, calendar support, and so forth, most participants used their phones only for making ordinary 

voice calls or for receiving/sending SMS. During the course of the first trial, students’ attitudes toward the services 

improved when they could for instance start choosing the channels of their preference and explore cost free the 

additional services. Through out both trials time periods the students perceived the MUSIS mobile services as 

something potentially useful, dynamic and as something that could be integrated in their every-day life. 

Did students find mobile phones and multicasting useful for supporting their learning? 

In the first set of trials, the participants were more likely to see the multicasting service as useful the more it was 

integrated into their course content. The cases presented in this paper differed substantially in how the instructors and 

the students used the technology. In the earlier trials, one instructor (for MEA708, in the School of Mathematics and 

Engineering) did not adapt his assignments or activities for the technology. The other instructor (for GIX 131, in the 

School of Humanities), actively produced content for this new medium. In addition to sending a relatively high 

number of MUSIS messages (41) to students, 7 were multimedia in form (video and audio). 

Table 1. Perceived usefulness of the educational mobile services after 5 weeks (n=41) and 10 weeks (n=41) 

Course Week Very Useful Fairly Not 

useful 

Useful Useful 

GIX131 5 27.3% 45.5% 18.2% 9 % 

MEA708 5 10.5% 52.6% 21.1% 15.8 % 

GIX131 10 40 % 26.7% 20% 13.3% 

MEA708 10 5.9% 35.3% 41.2% 17.6% 

The importance of integrating the service into the pedagogy or instructional style of the course is illustrated in Table 

1, which reports results from the survey item, “How useful did you experience the course related information sent to 

the educational channel”. In both classes, the majority of students saw the educational multicast services as useful or 

very useful in week 5. However, by week 10, that figure had dropped to less than 50% for MEA708. At the same 

time, the number of students in GIX131 viewing the services as “very useful” grew substantially. 

In the most recent trial conducted at BTH (END011), the initial thoughts of how to use the system between the 

students and the instructors differed. The students expressed interest in using the system for communication and 

relevant school information while the instructors’ concerns were on supplementing the student feedback with the 

technology. While the student groups sent 24 messages where 10 of them concerned specific project work scheduling 

and the remaining where more social. The instructors sent a high number of messages (25) where 12 were video 

feedback to the different groups about ongoing work and the remaining organizational about course times and 

deadlines. Table 2 illustrates how the perceptions changed as the system was used during the trial. Over the 5-weeks 

trial period, the students’ perceptions changed, team communication was used and perceived as being helpful and 

very helpful. The instructor’s feedback over the trial period was evaluated as being not helpful by more then a third 

of the students, while slightly less then a third felt more positive about the feedback and the remaining undecided. 

The analysis of these results gives us the opportunity to think about further research issues regarding how to best 

provide feedback with mobile devices in future trials. 

Table 2. Perceived usefulness of the educational mobile services for the END 011 course after 5 weeks (n=21) 

Initial Perceptions Week No interest Low interested Interested High Interested 

Team Communication 1 0.0% 19.0% 28.6% 52.4% 

Instructor Feedback 1 19.0% 33.3% 28.6% 19.0% 

Final Perceptions Week Not helpful Helpful Very helpful Undecided 

Team Communication 5 17.0% 21.0% 43.0% 19.0% 

Instructor Feedback 5 38.0% 5.0% 24.0% 33.0% 

68

With regard to usability and functionality of the phone itself, participants reported dissatisfaction with the small size 

of the mobile phone buttons, the quality of video, the small screen size, and the limited battery life of devices. 

Discussion 

These two trials clearly illustrate that both, students and teachers are open and intrigued while using everyday mobile 

communication and collaboration tools in education. What is still lacking is an understanding of how these tools 

provide new collaboration modes and how self-organizing environments can provide educational benefits (Dron, 

2007). The perceived needs of the instructors and students remain unsynchronized with the instructors’ desire to use 

the smart phones for providing feedback to the students while the students prefer more logistical and practical 

information to be delivered to the handsets. The creation of rich media like audio and video generated by the students 

requires more efforts than the traditional use of SMS and chat. From this perspective, having students working and 

communicating using these new media types may have some impact on the different educational activities (Lai & 

Wu, 2006). 

An unexpected finding took place in both trials based on the outcomes generated from the assignments developed by 

the instructor in the earlier pilot, and then again in the most recent trial regarding the use of the mobile application 

for many-to-many multicasting. Contrary to email, SMS, chat and other type of more instantaneous communication, 

students and instructors spent significant time staging and composing their answers and feedback, often recording 

multiple “takes” before the final video or audio was sent in to be multicasted. This suggests that a common practice 

of “composing” text messages may extend to audio and video messaging as well. Indeed, preliminary analysis of 

these recordings shows that the users tried to compress information not by indistinct, fast talk (similar to the 

abbreviations of SMS), but by concentrated, effectively expressed sentences. 

Conclusions and future development 

These studies were designed to explore the patterns of use of a number of mobile services experienced by students at 

a couple of university campuses and other locations of their choice. In the second trial these patterns of use were 

extended to provide the individuals the ability to multicast to the channels of their choice and to explore new patterns 

of collaboration. Impact on learning itself was not measured, nor would it have been possible to measure it 

meaningfully when the devices were used for such diverse purposes. Phone-optimized content was heavily used, and 

there was a clear request from the students that more resources be made available in this format, including 

administrative information from the universities. It is also important to recognize the need to address the technical 

requirements of producing and sharing of content across multiple types of devices and networks. This clearly points 

towards a low barrier for adoption of these mobile services by students in the near future if the ease of use between 

smart phones and traditional e-learning materials can effectively harnessed in ways that make sense and provided 

that the cost is comparable to wireless broadband. 

Ownership of the technology is clearly important. As long as the phones are loaned, students are reluctant to invest 

time and money in personalization. This will prevent better evaluation of the impact of technology on learning. 

Greater institutional support is needed in order for the smart phones to be used more fully. Regular updates of 

timetables and content, as well as adequate training and hardware provision are needed. As more students bring the 

technology with them to the university, change will most likely be driven by their demands as learners. 

Our results confirm the importance of designing applications and services for learners that are easy to use “on the 

road” that could be completed in short bursts of time (Wuthrich et al., 2003). Multicasting is one way to support what 

Brodersen, et al. (2005) call ‚”nomadic learners” who are more project oriented and who send much of their daily 

life‚ “transit between many physical places (“oasis”) such as classrooms, labs, workshops, libraries, museums, the 

city, nature, clubs and at home” (p. 298). However, our results also suggest that in higher education, a challenge is on 

designing for social technologies that allow for bridging different pedagogic goals (control of learning) and ways of 

communication between the different actors in the learning environment. These latest aspects require more than 

designing just services to connect people and content (Dron, 2007), but also creating new didactic sequences and 

educational activities that can connect formal and informal learning settings. 

69

As our work continues, we will try to enhance the educational aspects of the mobile services by developing and 

implementing various solutions to specific problems we have identified based on our observations and the data we 

have collected from the students. Our future efforts will continue to refine both the technology and activities for 

providing learners with more meaningful experiences with regard to the use of smart phones in educational settings 

by providing more tools for collaboration that take in consideration of the needs of both instructors and students. 

Coming research activities include the continuation of our efforts within the framework of a new international 

project exploring the use of mobile devices for game based learning and field studies in natural science, math, and 

physical fitness supported by mobile applications. 


MUSIS is partially funded by governmental bodies – Sweden’s VINNOVA and Israel 's MATIMOP - as part of the 

SIBED program, a joint Swedish-Israeli mobile technology research effort. 

References 

Brodersen, C., Christensen, B.G., Grønbæk, K. Dindler, C., & Sundararajah, B. (2005). eBag: A Ubiquitous Web 

Infrastructure for Nomadic Learning. Paper presented at the 14 th International conference on World Wide Web, May 

10-14, 2005, Chiba, Japan. 

Dron, J. (2007). Designing the Undesignable: Social Software and Control. Educational Technology & Society, 10 

(3), 60-71. 

Katz, J. E., & Aakhus, M. (2002). Perpetual Contact: Mobile Communication, Private Talk, Public Performance, 

Cambridge: Cambridge University Press. 

Lai, C.-Y., & Wu, C.-C. (2006). Using handhelds in a Jigsaw cooperative learning environment. Journal of 

Computer Assisted Learning, 22 (4), 284-297. 

Lankshear, C., & Knobel, M. (2006). New Literacies: Everyday Practices and Classroom Learning, Berkshire, UK, 


Milrad, M. (2003). Mobile Learning: Challenges, Perspectives and Reality. In K. Nyíri (Ed.), Mobile Learning: 

Essays on Philosophy, Psychology and Education, Vienna: Passagen Verlag, 25-38. 

Norris, C., Soloway, E., & Sullivan, T. (2002). Examining 25 years of technology in U.S. education. Communication 

of the ACM, 45 (8): 15-18. 

Roussos, G., Marsh, A.J., & Maglavera, S. (2005). Enabling Pervasive Computing with Smart Phones. IEEE 

Pervasive Computing, 4 (2), 20- 27. 

Tatar, D., Roschelle, J., Vahey, P., & Penuel, W. (2003). Handhelds Go to School: Lessons Learned. IEEE 

Computer, 36 (9), 30-37. 

Thornton, P., & Houser, C. (2004). Using Mobile Phones in Education. Paper presented at the 2 nd IEEE 

International Workshop on Wireless and Mobile Technologies in Education, 23-25 March 2004, Taoyuan, Taiwan. 

Varshney, U. (2002). Multicast over Wireless Networks. Communications of ACM, 45 (12), 31-37. 

Wuthrich, C., Kalbfleisch, G., Griffin, T., & Passos, N. (2003, April). On-line Instructional Testing in a Mobile 

Environment. Journal of Computing Sciences in Colleges, 18 (4), 23-29. 

70

Järvelä, S., Näykki, P., Laru, J., & Luokkanen., T. (2007). Structuring and Regulating Collaborative Learning in Higher Education 

with Wireless Networks and Mobile Tools. Educational Technology & Society, 10 (4), 71-79. 

Structuring and Regulating Collaborative Learning in Higher Education with 

Wireless Networks and Mobile Tools 

Sanna Järvelä, Piia Näykki, Jari Laru and Tiina Luokkanen 

University of Oulu, Finland // sanna.jarvela@oulu.fi // Fax +358 8 553 3744 

ABSTRACT 

In our recent research we have explored possibilities to scaffold collaborative learning in higher education with wireless 

networks and mobile tools. The pedagogical ideas are grounded on concepts of collaborative learning, including the 

socially shared origin of cognition, as well as self-regulated learning theory. This paper presents our three design 

experiments on mobile, handheld supported collaborative learning. All experiments are aimed at investigating novel 

ways to structure and regulate individual and collaborative learning with smartphones. In the first study a Mobile 

Lecture Interaction tool (M.L.I.) was used to facilitate higher education students’ self-regulated learning in university 

lectures. In the second study smartphones were used as regulation tools to scaffold collaboration by supporting 

externalization of knowledge representations in individual and collaborative levels. The third study demonstrates how 

face to face and social software integrated collaborative learning supported with smartphones can be used for 

facilitating socially shared collaboration and community building. In conclusion, it is stressed that there is a need to 

place students in various situations in which they can engage in effortful interactions in order to build a shared 

understanding. Wireless networks and mobile tools will provide multiple opportunities for bridging different contents 

and contexts as well as virtual and face to face learning interactions in higher education. 

Keywords 

Collaborative learning, Higher education, Mobile tools, Self-regulated learning, Wireless networks 

Introduction 

Recent developments in mobile technologies have contributed to the potential to support learners studying a variety 

of subjects (Scanlon, Jones & Waycott, 2005; Sharples, 2000) in elementary education (Zurita & Nussbaum, 2007) 

as well as in higher education (Baggetun & Wasson, 2006; Näykki & Järvelä, 2007; Milrad & Jackson, in press). 

Furthermore, there have also been efforts to improve the performance of knowledge workers in work-place settings 

(Brodt & Verburg, 2007). The integration of social software (web2.0) (Kolbitsch & Maurer, 2006; Cress & 

Kimmerle, 2007) and new mobile technologies (Kurti, Milrad & Spikol, 2007) has created interesting new 

possibilities for organizing novel learning and working situations. 

In higher education much effort has been made to find new ways to support individual student learning, but also to 

find ways for effective collaboration. Previous studies have explored how mobile technology can be used for offering 

an additional channel for the lecture interaction. The Classtalk project focused on giving lecturers the ability to pose 

questions to students (Dufresne, Gerace, Leonard, Mestre, & Wenk, 1996) while the eClass project provided 

facilities for structured capture and access of classroom lecture activities (Abowd, 1999). Ratto, Shapiro, Truong and 

Grisworld (2003) developed the ActiveClass application for encouraging lecture participation by using personal 

wireless devices. Furthermore, in order to support collaborative learning, new possibilities of mobile technologies 

have been explored. Interactive logbook (Chan, Corlet, Sharples, Ting & Westmanncott, 2005) provided the 

technology for knowledge sharing and multimedia notetaking, while many campus-wide laptop initiatives have 

provided students with access to social computing tools, such as instant messaging or chat (Gay, Stefanone, Grace- 

Martin, Hembrooke, 2001). 

Overall, the general claim has been that when new technologies and software have been used in an educational 

setting, new learning opportunities have arisen. Thus far there have been plenty of case studies and design 

experiments where mobile technologies have been used for innovative pedagogical ideas and design studies. 

However, only a few studies give detailed arguments as to what are these new opportunities in terms of learning 

interaction and collaboration and what are the exact processes that mobile tools can scaffold. We claim that it is not 

only the learner being “mobile” that matters. A stronger argument for applying mobile tools for education is that of 

increasing students’ opportunities for interactions and sharing ideas and thus, increasing opportunities for an active 

mind in multiple contexts (Dillenbourg, Järvelä, & Fischer, in press). In this paper we describe our current research 

focusing on exploring possibilities to scaffold collaborative learning in higher education with wireless networks and 

mobile tools. The pedagogical ideas are grounded on collaborative learning, including the socially shared origin of 






71

cognition as well as self-regulated learning theory. This is to say that special effort has been put on enhancing and 

scaffolding collaborative learning as cognitive, social, and motivated activity. 

Structuring and regulating collaborative learning 

Earlier research on collaborative learning has pointed out that shared understanding is not easy to achieve (Häkkinen 

& Järvelä, 2006; Leinonen, Järvelä & Häkkinen, 2005), and that students face difficulties engaging in learning and 

achieving their learning goals in a variety of learning contexts, including technology supported learning 

environments (Volet & Järvelä, 2001). In order to favour the emergence of productive interactions and to improve 

the quality of learning, different pedagogical models and technology-based regulation tools have been developed to 

support collaboration between participants. One way to enhance the process of collaboration, as well as to integrate 

individual and group-level perspectives of learning, is to structure learners’ actions with the aid of scaffolding or 

scripted cooperation (Fischer, Kollar, Mandl, & Haake, 2007). One of the ideas in this field is to design scripts that 

can be defined as “a set of instructions prescribing how students should perform in groups, how they should interact 

and collaborate and how they should solve the problem” (Dillenbourg, 2002, p. 63), that can be modified according 

to what kind of interaction, learning or outcomes are expected to be achieved. In scripted collaboration participants 

are supposed to follow prescriptions and engage in learning tasks. 

In addition to scripting as a mechanism to structure collaboration, we suggest that structuring can be enriched with 

technology-based regulation tools, which offer an individual and a group of learners opportunities to self-regulate 

their collaborative learning processes. Self-regulated learning theory concerns how learners develop learning skills 

and use learning skills effectively (Boekaerts, Pintrich & Zeidner, 2000). Self-regulated learners take charge of their 

own learning by choosing and setting goals, using individual strategies in order to monitor, regulate and control the 

different aspects influencing the learning process and evaluating his or her actions. Eventually, they become less 

dependent on others and on the contextual features in a learning situation. Although research into self-regulation has 

traditionally focussed on the individual perspective, there is increasing interest in considering the mental activities 

that are part of self-regulated learning at the social level, with reference to concepts such as social regulation, coregulation 

and shared regulation (McCaslin, 2004). 

Järvelä, Volet & Järvenoja (2005) characterize self-regulated learning from three perspectives. Self-regulated 

learning focuses on an individual as a regulator of a behavior and refers to the process of becoming a strategic 

learner by regulating their cognition, motivation and behavior to optimize learning (Schunk & Zimmerman, 1994). 

Conceptualizing self-regulated learning as a co-regulation has been influenced by socio-cultural theory and it 

emphasizes gradual appropriation of sharing common problems and tasks through interpersonal interaction (Hadwin, 

Wosney & Pontin, 2005; McCaslin & Hickey, 2001). The third perspective looks at how the regulation process can 

be framed to shared cognition and recent research on collaborative learning, which is in essence the co-construction 

of shared understanding (Roschelle & Teasley, 1995). This is collective regulation, where groups develop shared 

awareness of goals, progress, and tasks toward co-constructed regulatory processes, thereby sharing regulation 

processes as collective processes. 

Self-regulated learning theory has been used in our studies as a theoretical framework to develop those learning 

activities that give potential to individual and collaborative learning so that it stimulates active minds and 

interactions on individual and social levels. There are not yet studies which use mobile technology for supporting 

self-regulated learning, but recently there have been specific efforts made by people working on self-regulated 

learning theory to find ways to design technology to assist in helping students develop better learning strategies and 

regulate their learning process (e.g. Winne et al., 2006). 

Previous studies have explored how visualization as a form of regulation tool can be used for supporting individuals’ 

understanding (Larkin & Simon, 1987) or awareness of each others ideas (Fisher, Bruun, Gräsel, & Mandl, 2002; 

Leinonen & Järvelä, 2006). Visualizing individuals’ understanding can create for a group of learners a shared 

reference point, which supports focusing on central issues, for example shared or non-shared knowledge in a group’s 

interaction (Pea, 1994). Opportunities have been also searched from computer-based regulation tools, which aim to 

promote cognitive regulation processes. Learning tools are to promote motivated learning from the point of view of 

the individual learner as well as in opening new learning opportunities for social and interactive learning (Azevedo, 

2005). This developmental work can be used for compensating weak study and collaboration skills in different 

72

interactive learning environments, and studying in different domains. Azevedo and his colleagues (Azevedo, Guthrie 

& Seibert, 2004) have investigated the effects of goal-setting conditions on the ability of learners to regulate their 

learning in hypermedia environment. Their research results show that students use various types of self-regulatory 

behavior in learning with hypermedia, such as planning, monitoring, strategy use, task difficulty and demands and 

interest statements, but that students differ in their ability to regulate their learning. Later studies have put effort into 

designing computer-based scaffolds for self-regulated learning (Azevedo & Hadwin, 2005). For example, in a study 

that focused on collaborative planning and monitoring of students working within an online scientific inquiry 

learning environment, Manlove, Lazonder and de Jong (2006) examined the effect of a tool designed to support 

planning and monitoring in a scientific inquiry into the fluid dynamics on students’ model quality. The results 

showed a significant correlation between planning and model quality, indicating an overall positive effect for the 

support tool. 

Three studies of using wireless network and mobile tools in higher education 

In this paper our three design experiments on mobile, handheld supported collaborative learning are presented in 

order to demonstrate different pedagogical models and levels of scaffolds for socially-shared learning. Each 

experiment is aimed at investigating novel uses to structure and regulate collaborative learning with mobile tools. 

Mobile lecture interaction tool for activating students’ participation to the lecture interaction 

The aim of this study was to explore how the Mobile Lecture Interaction tool (M.L.I.) can be used for regulating and 

supporting students’ thinking and participation in the lecture interaction. It was studied how higher education 

students used the M.L.I. tool during lectures and in what ways the students view the M.L.I. tool as a support for their 

learning. 

Participating in the study were 173 higher education students (114 male and 59 female). The data were collected as a 

part of the authentic lecture situations in nine lectures where five lectures were in economics studies, two lectures in 

technical studies and two lectures in educational psychology studies. The lecture interaction was supported with the 

M.L.I.-tool, which was developed for this experiment (©Costa, 2006). The basic idea of the M.L.I.-tool is as follows: 

using personal, mobile devices (smartphones), students can anonymously ask questions, answer polls, and give 

feedback during the lecture (See Table 1). The tool allows every student and the lecturer to see these lists of 

questions. Furthermore, students have a possibility to vote on presented questions. Voting raises questions ranking in 

the display, encouraging the lecturer to give those questions precedence. 

Table 1. The M.L.I Pedagogical Structure 

Description of activities in the Lecture Pedagogical idea 

Send a question 

Encourage students’ cognitive activity and self-regulation 

in the lecture, engage students to the learning in the lecture 

Send a comment Enhance reflection 

Vote for a question or comment 

Enhance students’ metacognition and engage students’ 

learning in the lecture 

The data were collected via a questionnaire (including likert-scale questions as well as open ended questions), group 

interviews, lecture observations and log files. The process-oriented data, in a form of observation and log files, were 

collected in order to explore how students use M.L.I. tools as a part of their lecture interaction, e.g. what kind of 

questions or comments students present. The questionnaire and the interview data were collected to explore how 

students reflect the use of the M.L.I. tool. The results show that the students used the M.L.I. tool mostly for voting. 

The students reported that with the M.L.I. tool they were more active in thinking of questions and evaluating the 

presented questions’ meaning for themselves than they normally are during the lectures. Furthermore, the use of the 

M.L.I. tool supported students’ feelings of belonging to a group. The students mentioned that the use of the M.L.I 

tool supported their engagement in the content of the lecture. Their concentration did not stray as much as it did in 

lectures. Awareness of other students’ questions offered new ideas for the students and therefore that was seen as 

valuable for their learning. 

73

Mobile mind map tool for stimulating collaborative knowledge construction in groups 

The aim of this study was to investigate the process of collaborative knowledge construction when technology and 

self-generated pictorial knowledge representations are used for visualizing individual and group shared ideas (See 

Näykki & Järvelä, 2007). In particular, the aim was to find out how students contribute to the group’s co-regulation 

of collaborative knowledge construction and use each other’s ideas and cognitive tools as a provision for their jointly 

evolving cognitive systems. 

The participants of this study were teacher education students (N = 13, 5 male, 8 female) who were randomly 

assigned to work in groups. Their working was scaffolded with the Mobile Mind Map tool (© Scheible, 2005, see 

Figure 1.), and a problem-oriented pedagogical structure. Student activities were structured around different phases 

in which they brainstormed, explored real-life examples to visualize their thinking, and used pictures as knowledge 

representations to answer to the learning task. The Mobile Mind Map tool allowed students to take pictures with a 

smart phone and to add text annotations to the pictures. The annotated pictures were sent to the server and they were 

used to construct a mind map with the computer (See Table 2). 

Table 2. “The Mobile mind map” pedagogical structure 

Description of group activities Pedagogical idea Outcome 

1. Mind mapping with paper and pen Grounding Mind map with paper & pen 

2. Campus area exploration for evidence 

with mobile phones 

Inquiring Annotated pictures 

3. Mind mapping with Mind Map Tool and 

pictures by the laptop computer 

Constructing Mind map with pictures 

4. Reflection on the experience Reflecting Shared experiences 

Figure 1. The Mind map tool 

The process-oriented data involved students’ videotaped face-to-face group activities (mind-mapping with paper and 

pen and mind-mapping with a Mobile Mind Map tool and pictorial knowledge representations) as well as stimulated 

74

ecall interviews (Ericsson & Simon, 1980). The data-driven qualitative content analysis revealed that pictures as 

self-generated knowledge representations were used for carrying individuals’ abstract meanings. Furthermore, 

students’ activity in processing each others ideas further, as well as a level of cognitively challenging activities, 

indicates that pictorial knowledge representations and technology tools scaffolded co-regulation of collaborative 

knowledge construction. However, students were of the opinion that pictorial knowledge representations are 

challenging and thorough negotiation is needed for grounding pictures on the content discussions. 

Mobile “Edufeeds” for creating shared understanding among virtual learning communities 

The aim of this study was to explore how mobile technologies and social software (weblogs, wikis, RSS-aggregators 

and file-sharing services) can be used for scaffolding collaborative learning; sharing understanding and building 

virtual communities (See Näykki & Järvelä, 2007). The main assumption was that students’ interactions will be 

enriched when possibilities of social software are integrated in the learning situations, and thus, building virtual 

communities will be more fluent than in more traditional virtual learning environments. 

The participants of this study (N = 22, 5 male and 17 female) worked in groups of 4-5 students for a period of three 

months. Group work was structured to different phases, which were facilitated with social software (weblog, wikis 

and RSS-aggregators) as well as mobile phones and lap top computers (See Table 3). Visualizing ideas with pictorial 

knowledge representations was a tool for students to plan and monitor their individual level and group level learning 

processes. 

Description of 

activities 

Lecture 

(6 sessions) 

Face to face group 

working sessions 

(6 sessions) 

Individual working 

sessions 

(6 sessions) 

Group’s meaning 

making sessions 

(2 sessions) 

Virtual group 

working sessions 

(2 sessions) 

Table 3. Pedagogical structure 

Pedagogical idea Outcome Technological tool 

Theoretical 

grounding 

Theoretical concepts - 

Groups’ grounding Shared concepts - 

Constructing, 

monitoring 

Elaborating 

Constructing, 

monitoring 

Constructing 

knowledge 

representations 

Negotiating 

knowledge 


Using knowledge 


Mobile phone, 

Weblog, 

RSS-aggregator 

Weblog 

Wiki 

Weblog, 

Wiki 

RSS-aggregator 

The students were introduced to the content of the course with six lectures and after the each lecture the students 

reflected on the content of the lectures in groups. The given task for each group was to first reflect on the content and 

to name five important themes in the lecture. After that the students were asked to choose one of the themes and to 

formulate their group’s working problem based on the theme with which they continued to work by finding real-life 

examples to represent their shared discussions. In practise, the group work was followed with a one-week phase of 

independent on-line work, where students were asked to use mobile phones to take pictures and/or video clips to 

represent their ideas of the learning content. While taking the pictures/videos students were also asked to answer the 

following questions: what is the name of this picture, what does this picture represent, and how is it related to the 

learning content, by typing a short description for the picture. The pictures and descriptions of the pictures were sent 

automatically to the each student’s own weblog. This same task continued after each lecture. Weblogs were used as 

personal journals, where students reflected their ideas further by writing journal entries around the respective 

pictures/videos. Furthermore, students were asked to follow each others’ contributions to their personal weblogs by 

using RSS-aggregators in their mobile phones. In the middle of the course, students had a “meaning-making session” 

where they reviewed all the group members’ weblogs to see the pictorial material everyone had collected. The 

students were asked to introduce the pictures by explaining what they represent and to negotiate and choose among 

75

the pictorial knowledge representations those pictures that could represent the group’s shared understanding. This 

session was repeated twice, in the middle of the course and at the end of the course, and the session was followed by 

the virtual group working phase, where students continued to share their ideas with the chosen pictorial knowledge 

representations. 

The data were collected by using video-observation, questionnaire, stimulated recall interviews. In addition log files 

of the students’ activities, and students’ and groups’ products in weblogs and wikis were used as a data. The results 

showed that construction and sharing of knowledge representations scaffold students’ shared regulation of 

collaborative learning by activating their cognitive processes; explaining and elaborating their own understanding. 

Conclusions 

In this paper three different studies were presented in order to illustrate how collaborative learning can be structured 

and regulated in higher education, with wireless networks and mobile tools. The pedagogical ideas derive from 

collaborative learning, including the socially shared origin of cognition as well as self-regulated learning theory. 

Each study shows that, with a novel use of technology, certain aspects of collaboration can be supported. 

The Mobile Lecture Interaction Tool (M.L.I) is an example of how individual self-regulation can be supported in 

university lectures. The results show that students’ cognitive activities, such as metacognition and reflection, were 

stimulated by focusing on questions about the content of the lecture. This kind of learning tool can be used for 

compensating weak study skills in different domains (Azevedo, Guthrie & Seibert, 2004) and especially in university 

lectures. Even though, the M.L.I. tool was credited as beneficial for lecture interaction, some students viewed that 

there were too many things to concentrate on at the same time. Students pointed out that listening to the lecturer, 

writing lecture notes and using the M.L.I. tool at the same time, were too much for them to handle. Therefore, 

aspects related to cognitive load (Kirschner, 2002) need to be recognized. To lower the cognitive load the M.L.I. tool 

could be developed further so that students and teacher could use questions and comments also after the lecture. 

Another possibility is structuring or scripting the lecture situation so that there are specific times for questions and 

comments, and therefore students would not feel that they are missing some important things while they are using the 

M.L.I. tool. 

The results of the Mobile Mind Map tool study imply that the tool can be used for enhancing co-regulation in terms 

of sharing and externalizing visual knowledge representations and developing them in interpersonal interactions. The 

students should be encouraged, when appropriate, to create, modify or co-design the learning environments in which 

they are working. Construction of external representations can be a successful learning strategy that not only helps 

students by externalizing and sharing their thoughts, but also provides a source of information for teachers about 

students’ current task understanding (Butler & Cartier, 2004). Students’ co-regulated learning can be supported by 

scaffolding externalization of their own thinking and by seeing what others are thinking and, when possible, to then 

continue their own and others’ flow of thinking. However, externalizing knowledge representations is cognitively 

challenging. In this study pictures were used as external knowledge representations and the study indicates that 

pictures can carry individuals’ co-created meanings in them but since those meanings are personal, ambiguous 

negotiation processes are highly valuable. 

The Edufeed study indicates that web 2.0 technologies can be designed to scaffold shared regulation processes within 

the groups. In the study the students constructed a variety of pictorial knowledge representations and shared their 

meaning collaboratively. However, the students thought that it was challenging to represent their own ideas with 

pictures and even more challenging to explain the representation to others. Nevertheless, the students who explained 

their pictures to others viewed that the elaboration of pictures was beneficial for their learning. The results showed 

that construction and sharing of knowledge representations activated students' self-regulated learning; explaining and 

elaborating their own understanding. Other studies from collaborative learning contexts have highlighted that in 

collaboration, individuals should share their understanding in discussion to converge their knowledge 

representations, which might lead to new shared knowledge representations (e.g. Dillenbourg & Traum, 2006; Fisher 

& Mandl, 2005; Rochelle, 1992). It is concluded that even though self-generated pictorial knowledge representations 

are often personal and ambiguous they may give an impetus for sharing and explaining individual knowledge 

representations to the socially shared level. 

76

In conclusion, this paper stresses a need to place students in various situations in which they can engage in effortful 

interactions in order to create opportunities for active minds (Dillenbourg, Järvelä, & Fisher, in press). Wireless 

networks and mobile tools will provide multiple opportunities for bridging different contents and contexts, as well as 

virtual and face to face learning interactions in higher education. Since the learning environment in higher education 

is more open and less teacher-guided than at other educational levels there is a need to increase student opportunities 

for self-regulating their learning on an individual as well as socially shared level. Wireless networks and mobile tools 

will provide future potential for developing learning in higher education, which needs to be explored in detail. 

References 

Abowd, G. D. (1999). Classroom 2000: an experiment with the instrumentation of a living educational environment. 

IBM Systems Journal, 38 (4), 508–530. 

Azevedo, R. (2005). Computers as metacognitive tools for enhancing learning. Educational Psychologist, 40 (4), 

193-197. 

Azevedo, R., & Hadwin, A. (2005) Scaffolding self-regulated learning and metacognition – implications for the 

design of computer-based scaffolds. Instructional Science, 33 (5-6), 367-379. 

Azevedo, R., Guthrie, J.T., & Seibert, D. (2004). The role of self-regulated learning in fostering students’ conceptual 

understanding of complex systems with hypermedia. Journal of Educational Computing Research, 30 (1), 87-111. 

Baggetun, R., & Wasson, B. (2006). Self-Regulated Learning and Open Writing. European Journal of Education, 41 

(3-4), 453-472. 

Boekarts, M., Pintrich, P. R., & Zeidner, M. (2000). Handbook of self-regulation, San Diego, CA: Academic Press. 

Brodt, T., & Verburg, R. (2007). Managing mobile work – insights from European practice. New Technology, Work 

and Employment, 22 (1), 52-65. 

Butler, D. L., & Cartier, S. C. (2005). Multiple complementary methods for understanding self-regulated learning as 

situated in context. Paper presented at the Annual Meeting of the American Educational Research Association, April 

11-15, 2005, Montreal, Canada. 

Chan, T., Corlett, D., Sharples, M., Ting, J., & Westmancott, O. (2005). Developing interactive logbook: a personal 

learning environment. Paper presented at the 2005 IEEE International Workshop on Wireless and Mobile 

Technologies in Education, November 28-30, 2005, Tokushima, Japan. 

Cress, U., & Kimmerle, J. (2007). A theoretical framework of collaborative knowledge building with wikis – a 

systemic and cognitive perspective. Paper presented at the 7 th International Computer Supported Collaborative 

Learning Conference, July 16-21, 2007, New Brunswick, NJ, USA. 

Dillenbourg, P., Järvelä, S., & Fischer, F. (in press). The evolution of research on computer-supported collaborative 

learning: from design to orchestration. To appear in Kaleidoscope Legacy Book. 

Dillenbourg, P., & Traum, D. (2006). Sharing solutions: persistence and grounding in multimodal collaborative 

problem solving. The Journal of the Learning Sciences, 15 (1), 121–151. 

Dillenbourg, P. (2002). Over-scripting CSCL: The risks of blending collaborative learning with instructional design. 

In P. A. Kirschner (Ed.), Three worlds of CSCL. Can we support CSCL,. Heerlen: Open Universiteit Nederland, 61- 

91. 

Dufresne, R. J., Gerace, W. J., Leonard, W. J., Mestre, J. P., & Wenk, L. (1996). Classtalk: A classroom 

communication system for active learning. Journal of Computing in Higher Education, 7 (2), 3-47. 

77

Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87 (3), 215-251. 

Fischer, F., Kollar, I., Mandl, H., & Haake, J. M. (2007). Scripting computer-supported collaborative learning – 

cognitive, computational, and educational perspectives, New York: Springer. 

Fischer, F., & Mandl, H. (2005). Knowledge convergence in computer-supported collaborative learning: the role of 

external representation tools. The Journal of the Learning Sciences, 14 (3), 405–441. 

Fischer, F., Bruun, J., Gräsel, C., & Mandl, H. (2002). Fostering collaborative knowledge construction with 

visualization tools. Learning and Instruction, 12 (2), 213-232. 

Gay, G., Stefanone, M., Grace-Martin, M., & Hembrooke, H. (2001). The Effects of Wireless Computing in 

Collaborative Learning Environments. International Journal of Human-Computer Interaction, 13 (2), 257-276. 

Hadwin, A.F., Wozney, L., & Pontin, O. (2005). Scaffolding the appropriation of self-regulatory activity: A sociocultural 

analysis of changes in student-teacher discourse about a graduate research portfolio. Instructional Science, 

33 (5-6), 413-450. 

Häkkinen, P., & Järvelä, S. (2006). Sharing and constructing perspectives in web-based conferencing. Computers 

and Education, 47 (4), 433-447. 

Järvelä, S. , Volet, S. & Järvenoja, H. (2005). Motivation in collaborative learning: New concepts and methods for 

studying social processes of motivation. A paper presented at the Earli 2005 conference, 22-27 August 2005, 

Nicosia, Cyprus. 

Kirschner, P. A. (2002). Cognitive load theory. Learning and Instruction, 12 (1), 1-10. 

Kolbitsch, J., & Maurer, H. (2006). The Transformation of the Web: How Emerging Communities Shape the 

Information We Consume. Journal of Universal Computer Science, 12 (2), 187-213. 

Kurti, A., Milrad, M., & Spikol, D. (2007). Designing Innovative Learning Activities Using Ubiquitous Computing. 

Paper presented at the ICALT 2007 conference, July 18-20, 2007, Nagaoka, Japan. 

Larkin, J.H., & Simon, H.A. (1987). Why a diagram is (sometimes) worth of ten thousand words. Cognitive Science, 

11 (1), 65-99. 

Leinonen, P., & Järvelä, S. (2006). Facilitating interpersonal evaluation of knowledge in a context of distributed 

team collaboration. British Journal of Educational Technology, 37 (6), 897-916. 

Leinonen, P., Järvelä, S., & Häkkinen, P. (2005). Conceptualizing the awareness of collaboration: A qualitative study 

of a global virtual team. Computer Supported Cooperative Work, 14 (4), 301-322. 

Manlove, S., Lazonder, A.W., & De Jong, T. (2006). Regulative support for collaborative scientific inquiry learning. 

Journal of Computer Assisted Learning, 22 (2), 87-98. 

McCaslin, M. (2004). Coregulation of opportunity, activity, and identity in student motivation. In D. McInerney & S. 

Van Etten (Eds.), Big theories revisited: Research on sociocultural influences on motivation and learning, 

Greenwich, CT: Information Age, 249-274. 

McCaslin, M., & Hickey, D. T. (2001). Self-regulated learning and academic achievement: A Vygotskian view. In B. 

Zimmerman & D. Schunk (Eds.), Self-regulated learning and academic achievement: Theory, research, and 

practice, Mahwah, NJ: Lawrence Erlbaum, 227-252. 

Milrad, M., & Jackson, M. (in press). Designing and Implementing Educational Mobile Services in University 

Classrooms Using Smart Phones and Cellular Networks. International Journal of Engineering Education. 

78

Näykki, P., & Järvelä, S. (2007). Pictorial Knowledge Representations and Technology Tools for Regulating 

Collaborative learning. Paper presented at the 12 th Biennial Conference for Research on Learning and Instruction, 

August 28 -Septermber 1, 2007, Budapest, Hungary. 

Pea, R.D. (1994). Seeing what we build together: Distributed multimedia learning environments for transformative 

communications. The Journal of the Learning Sciences, 3 (3), 285-299. 

Ratto, M., Shapiro, R. B., Truong, T. M., & Griswold,W. G. (2003). The ActiveClass Project: Experiments in 

Encouraging Classroom Participation. Paper presented at the CSCL 2003 Conference, June 14-18, 2003, Bergen, 

Norway. 

Roschelle, J., & Teasley, S. (1995). The construction of shared knowledge in collaborative problem solving. In C. E. 

O’Malley (Ed.), Computer supported collaborative learning, Heidelberg: Springer, 69-97. 

Roschelle, J. (1992). Learning by collaborating: convergent conceptual change. The Journal of the Learning 

Sciences, 2 (3), 235–276. 

Scanlon, E., Jones, A., & Waycott, J. (2005). Mobile technologies: prospects for their use in learning in informal 

science settings. Journal of Interactive Media in Education, 25, retrieved October 15, 2007, from 

http://jime.open.ac.uk/2005/25/scanlon-2005-25.pdf. 

Schunk, D. H., & Zimmerman, J. (1994). Self-regulation of learning and performance. Issues and educational 

applications, Hillsdale, NJ: Erlbaum. 

Sharples, M. (2000). The design of personal mobile technologies for lifelong learning. Computers & Education, 34 

(3-4), 177–193. 

Volet, S. E., & Järvelä, S. (2001). Motivation in learning contexts: Theoretical advances and methodological 

implications, Amsterdam: Elsevier Science. 

Winne, P. H., Nesbit, J. C., Kumar, V., Hadwin, A. F., Lajoie, S. P., Azevedo, R. A., & Perry, N. E. (2006). 

Supporting self-regulated learning with gStudy software: The Learning Kit Project. Technology, Instruction, 

Cognition and Learning, 3 (1), 105-113. 

Zurita, G., & Nussbaum, M. (2007). A constructivist mobile learning environment supported by a wireless handheld 

network. Journal of Computer Assisted Learning, 20 (4), 235-243. 

79

Al-A'ali, M. (2007). Implementation of an Improved Adaptive Testing Theory. Educational Technology & Society, 10 (4), 80-94. 

Implementation of an Improved Adaptive Testing Theory 

Mansoor Al-A'ali 

Department of Computer Science, College of Information Technology, University of Bahrain, Kingdom of Bahrain, 

malaali@itc.uob.bh // mansoor.alaali@gmail.com 

ABSTRACT 

Computer adaptive testing is the study of scoring tests and questions based on assumptions concerning the 

mathematical relationship between examinees’ ability and the examinees’ responses. Adaptive student tests, 

which are based on item response theory (IRT), have many advantages over conventional tests. We use the least 

square method, a well-known statistical method, to reach an estimation of the IRT questions’ parameters. Our 

major goal is to minimize the number of questions in the adaptive test in order to reach the final level of the 

students’ ability by modifying the equation of estimation of the student ability level. This work is a follow-up on 

Al-A'ali (2007). We consider new factors, namely, initial student ability, subject difficulty, number of exercises 

covered by the teacher, and number of lessons covered by the teacher. We compared our conventional exam 

results with the calculated adaptive results and used them to determine IRT parameters. We developed the IRT 

formula of estimating student ability level and had positive results in minimizing the number of questions in the 

adaptive tests. Our method can be applied to any subject and to school and college levels alike. 

Keywords 

IRT, Item response theory, Testing methods, Adaptive testing, Student assessment 

Introduction 

Item response theory (IRT) is the study of scoring tests and questions based on assumptions concerning the 

mathematical relationship between the examinee’s ability (or other hypothesized traits) and the questions’ responses. 

Adaptive student tests, which are based on IRT, have many advantages over conventional tests. The first advantage 

is that IRT adaptive testing contributes to the reduction of the length of the test because the adaptive test gives the 

most informative questions when the student shows a mastery level in a certain field. Secondly, the test can be better 

tailored for individual students. 

Adaptive assessment would undoubtedly present improved methods of assessment, especially with the availability of 

computers in all schools. As we know, evaluation and assessment are an integral part of learning. A good objective 

test at the end of each learning objective can reveal a great deal about the level of understanding of the learner. The 

possibility of differential prediction of college academic performance was discussed by researchers (Young, 1991). 

A good analysis of item response theory was presented in (Fraley, Waller, & Brennan, 2000) who applied the theory 

to self report measures of adult attachment. Error-free mental measurements resulting from applying qualitative item 

response theory to assessment and program validation, including a developmental theory of assessment, were 

discussed in Hashway (1998). The general applicability of item response models was extensively discussed by Stage 

(1997a, 1997b, 1997c, & 1997d). The use of the item response theory for the issue of gender bias in predicting 

college academic performance was discussed in Young (1991). Some researchers proposed implementing decision 

support systems for IRT-based test construction (Wu, 2000). The IRT algorithm aims to provide information about 

the functional relation between the estimate of the learner’s proficiency in a concept and the likelihood that the 

learner will give the correct answer to a specific question (Gouli, Kornilakis, Papanikolaou, & Grigoriado, 2001). 

Figure 1 is a schematic representation of an adaptive test. 

The IRT algorithm illustrated in Figure 1 aims to provide information about the functional relation between the 

estimate of the learner’s proficiency on a concept and the likelihood that the learner will give the correct answer to a 

specific question (Gouli et al., 2001). In a conventional test, two matters are considered: the time and the length of 

the test. In an adaptive test, possible termination criteria are: 

1. The number of questions posed exceeds the maximum number of questions allowed. 

2. The accuracy of the estimation of the learner’s proficiency reaches the desired value. 

3. Time limitations: most popular adaptive tests have a time limit. Although time limitation is not necessary in 

adaptive testing, it can be beneficial. Students who spend too much time on tests may get tired, which can 

negatively affect the score. 






80

4. No more relevant items in the item bank: when the item bank is small, or questions with a difficulty level 

suitable for the student do not exist, the test must be terminated. 

Figure 1. A schematic representation of an adaptive test 

Conventional exams suffer from certain problems that must be considered carefully. First, assessments and 

assignments are normally given to test the different capabilities of students who are in the range of poor to excellent, 

and thus all students have to meet or resolve the same standard questions, which mostly do not match their own 

capabilities, and are either lower or higher. This means that students with high capabilities may waste their time in 

solving average assignments that do not excite, challenge, or interest them. Another aspect that we should keep in 

mind considerably is that a score of 75% on a test containing easy items has a different meaning from a score of 75% 

on a test containing difficult items (Segall, 2000). Second, testing for knowledge and understanding in the context of 

a specific course typically involves administering the same set of test items or questions to all the students enrolled in 

the course, usually at the same examination sitting. 

When we consider classical test theory (CTT), we realize that CTT has a number of deficiencies. One of the 

problems with CTT is that it is test-oriented rather than item-oriented; that is, in the classical true-score model there 

is no regard for an examinee’s response to a given item. As a result, CTT does not allow predictions to be made 

about how an individual or group of examinees will perform on a given item (Hildegarde & Jacobson, 1997). 

To formulate a complete understanding of the assessment process in schools, we look at some drawbacks of penciland-paper 

tests. Some drawbacks of conventional pencil-and-paper tests are scoring and feedback. Instructors need a 

lot of time to correct test papers. This means that examinees cannot be informed about the result of the test 

immediately after its completion. We know that immediate feedback is important to the students for psychological 

reasons. It is motivational, helps them focus, and informs them if they have to work harder. 

One of the solutions to these problems is to use computer adaptive tests (CATs). Advantages of CATs can include 

shorter and quicker tests, flexible testing schedules, increased test security, better control of item exposure, better 

balancing of test content areas for all ability levels, quicker test-item updating, quicker reporting, and a better testtaking 

experience for the test-taker. CATs are widely used these days and they give good results in many educational 

fields. CATs are used in many professional certification programs. Novell successfully introduced CATs into its 

certification program in 1991. The Educational Testing Service, the world’s largest testing organization, published 

the Graduate Record Exam (GRE) as an adaptive test in 1993. TOEFL also uses a CAT. The Nursing Boards 

converted completely from paper-based testing to a computerized adaptive test in 1994. 

Assessments can guide improvement—presuming they are valid and reliable—if they motivate adjustments to the 

educational system (Shute & Towle, 2003). “The question is no longer whether assessment must incorporate 

technology. It is how to do it responsibly, not only to preserve the validity, fairness, utility, and credibility of the 

measurement enterprise but, even more so, to enhance it” (Bennett & Persky, 2002). 

Intelligent tutoring systems permit the modeling of an individual learner, and with that modeling comes the 

knowledge of how to perform an individualized assessment. Examinations can once again be tailored to meet the 

individual needs of a particular learner. In contrast with paper-and-pencil multiple-choice tests, new assessments for 

81

complex cognitive skills involve embedding assessments directly within interactive, problem-solving, or open-ended 

tasks (Bennett & Persky, 2002). 

One of our main objectives in this research was to study IRT in order to reduce the number of questions before 

reaching stability. It had been shown (Gouli et al., 2001; Eggen & Straetmans, 2000) that after 13–15 questions, the 

level of students’ ability becomes stable. Therefore, the length of time required to complete the adaptive test is 

shorter than that required for the pencil-and-paper test. Our hypothesis is that the starting level of the adaptive test is 

arbitrary. Our starting point in the adaptive test was at the conventional level that we got from stage 1 of the method 

previously mentioned. Our target was to get a stable level after seven questions only, hence reducing the length of 

the test and the time required to complete the test. In order to test our hypothesis we had to build a system and 

evaluate and enhance IRT. We measured the effectiveness of IRT by comparing the students’ achievement levels 

using the new IRT-based system with the results achieved by the students after taking an ordinary written test 

prepared by the teachers. 

Testing, Adaptive Testing, and Item Response Theory 

Linear tests are not adaptively administered tests and thus they are presented on the far-left side of Figure 2. Items on 

these tests are represented in sequence, that is, they are linear. The examinee is presented with the first item, then the 

second, then the third, and so on, in a predetermined fashion. Linear tests administered on the computer are also 

known as fixed-form tests. 

In linear-on-the-fly testing (LOFT), unique, fixed-length tests are constructed for each examinee. The target of 

content and psychometric specifications should be met in constructing the test. However, the examinee’s proficiency 

level is not a consideration when constructing form items; thus, these tests are not adaptive. A large pool of items is 

needed to develop this type of test because the test forms should be unique. The benefits of constructing LOFT tests 

are in presenting item exposure and rigorous content-ordering requirements. Therefore, the LOFT model has one 

major advantage over the linear model: improved security that comes from presenting different items across forms. 

Figure 2. Types of Testing 

Testlets are groups of items that are considered a unit and administered together. Usually, testlets are constructed 

based on previous knowledge of the difficulty of the items or based on content. More specifically, testlets are 

developed according to the order of items’ difficulty or their ability to meet content specifications. Testlets are 

presented to examinees in units. Within testlets, examinees are given the opportunity to review, revise, and omit 

items. Items within a testlet may be assembled by similarities in level of difficulty, subject matter, or both. Hence, 

multistage testing will be possible. 

Mastery Models tests are developed to provide accurate information about mastery/non-mastery. The main goals of 

mastery models are (1) covering the content domain and (2) making accurate mastery decisions. There are various 

models to implement mastery models, and they all share a major advantage, namely efficiency. Efficiency is 

observed in the ease of classifying all examinees based on simple rules. Eggen and Straetmans (2000) and Rudner 

(2002) provide good descriptions of classifying examinees into three categories. 

Tests based on CAT delivery models present items depending on the performance of the examinee. The items that 

are presented have been pre-tested and item parameter estimates have been calculated. Using this information, 

examinees receive items that match their proficiency level at that time. Adaptive assessment systems can ask the 

most informative questions and determine when the student has displayed mastery of a particular concept. There is 

no need to pose further test items. The test may be better tailored for individual students. 

82

Adaptive assessment has two main goals. First, the length of the test may be decreased because the adaptive test 

gives the most informative questions when the student shows a mastery level in a certain field. Second, the test may 

be better tailored for individual students. Adaptive assessment can provide accurate estimation of the learner’s 

proficiency in an efficient way without forcing him/her to answer questions that are either too easy or too difficult for 

him/her (Gouli et al. 2001). 

In the example shown in Figure 3, the examinee’s level was about 50 out of 100, i.e., his capability for answering 

questions of varying difficulty is average. The first question the computer gave the examinee was of level 52, i.e., a 

slightly more difficult question than his initial estimated capability. The examinee correctly answered the first two 

questions and his level rose. When he answered the third question incorrectly, his level went down. The process 

continued until there was minimal error in estimating the examinee’s level. The computer program becomes more 

and more certain that the examinee’s ability level is close to 50 (Linacre, 2000). 

Figure 3. Example CAT Test Administration 

In this research project, we use item response theory (IRT). IRT is a modern test theory designed to address the 

shortcomings inherent in classical test theory methods for designing, constructing, and evaluating educational and 

psychological tests (Hambleton, Swaminathan, & Rogers, 1991). One of the functions of IRT plots respondent 

behavior across the continuum of abilities underlying the concept of interest. In other words, IRT is adaptive. 

IRT was used in the implementation of adaptive assessments because the proficiency estimate is perhaps independent 

of the particular set of questions selected for the assessment. Each learner gets a different set of questions with 

different difficulty levels while taking the adaptive assessment (Weiss, 1983). IRT-based item selection strategies 

(Weiss, 1983) are maximum information item selection strategy, and Bayesian item selection strategy. 

Kingbury and Weiss used maximum information item selection strategy, in which the item pool is searched for an 

item that can give maximum information about the examinee. In Bayesian item selection strategy, used by McBride 

and Martin, the item selected from the item pool will maximally reduce the posterior variance of an individual’s 

ability estimate. Bayesian item selection strategy uses prior information about the examinee more completely than 

maximum information item selection strategy (Weiss, 1983). 

Eggen and Straetmans (2000) showed that optimum item selection is largely responsible for the major efficiency 

gains of CAT against a fixed, linear paper-and-pencil test. Efficiency gains means that fewer items are required to 

assess candidates with the same degree of accuracy, or that they can be assessed much more accurately with the same 

number of items. Table 1 shows that compared to pencil-and-paper tests, CAT achieves a higher percentage of 

correct answers when using maximum information, and fewer items are required to achieve a similar level of 

accuracy. 

83

Table 1. CAT and a linear mathematics intake test 

Method Average number of items % correct decisions 

Paper-and-pencil 

25 

87.0 

CAT(maximum information) 

14.2 

88.3 

CAT (random selection) 

20.2 

85.2 

Adaptive Assessment Algorithm 

The IRT algorithm aims illustrated in Figure 1 provide information about the functional relation between the 

estimate of the learner’s proficiency in a concept and the likelihood that the learner will give the correct answer to a 

specific question (Gouli et al., 2001). 

Item Characteristic Curve in IRT 

The item characteristic curve (ICC) is the basic building block of item response theory. There are two technical 

properties of an item characteristic curve. The first is the difficulty of the item. Under item response theory, the 

difficulty of an item describes where the item functions along the ability scale. The second technical property is 

discrimination, which describes how well an item can differentiate between examinees with abilities below the item 

location and those with abilities above the item location (Baker, 2001). 

At each ability level, there will be a certain probability that an examinee with that ability will give a correct answer 

to the item. This probability will be denoted by P(θ). Its formula is given by 

1 1 

P( 

θ) 

= = 

− L −a( 

θ−b) 

1+ 

e 1+ 

e 

where: 

b is the difficulty parameter 

a is the discrimination parameter 

L = a(θ - b) is the logistic deviate (logit) & 

θ is an ability level. 

The importance of item discrimination comes from the fact that it relates the strength of the relationship between a 

test item and the underlying (and unobservable) attribute being measured, for example, knowledge or learning. 

In the case of a typical test item, this P(θ) will be small for examinees of low ability and large for examinees of high 

ability (Baker, 2001). Adding one more factor to the previous factor, the formula will be (Gouli et al., 2001): 

1−c 

P( 

θ ) = c + 

−2( 

θ−b 

) 1+ 

e 

where c is unknown. Notice that a = 2 is given for the purpose of simplification. 

Item Information Function 

Item information function (IIF) is considered a very important value in IRT. IFF is used in estimating the value of the 

ability parameter for an examinee. Moreover, it is related to the standard deviation of the ability estimation. If the 

amount of information is large, it means that an examinee whose true ability is at that level can be estimated with 

precision; that is, all the estimates will be reasonably close to the true value. If the amount of information is small, it 

means that the ability cannot be estimated with precision, and the estimates will be widely scattered about the true 

ability (Baker, 2001). 

In statistics, Sir R. A. Fisher defined information as the reciprocal of the precision with which a parameter could be 

estimated (Baker, 2001). Statistically, the precision with which a parameter is estimated is measured by the 

variability of the estimates around the value of the parameter. The amount of information is given by the formula: 

1 

I 

= 

σ 2 

84

where σ 2 is a measure of precision of the variance of the estimators. 

Estimating Item Parameters 

I) Rasch model for estimating item parameters 

Mathematical analysis shows that the Rasch model is statistically strong. Rasch model estimates are sufficient, 

consistent, efficient, and unbiased. It estimates the difficulty parameter and student ability (student level). There is an 

efficient method for approximating parameter estimates that could easily be calculated by hand. The drawback of this 

model is that there are no guessing factors and discrimination parameters. Besides, estimating items’ parameters and 

students’ abilities in a test with 20 or 30 items requires at least 100 examinees. 

II) Chi-square goodness-of-fit 

When the conventional test finishes, we observe a sample of M examinees’ responses to N items in the exam. 

According to examinees’ achievement results, we determine the examinees’ ability levels. These ability levels will 

be distributed over the ability scale. According to Baker, “The agreement of the observed proportions of correct 

response and those yielded by the fitted item characteristic curve for an item is measured by the chi-square goodnessof-fit 

index” (2001). 

III) Method of least square 

This method works when the regression is approximately linear. The equation of a straight line is determined by the 

points that it passes through. 

IV) Level estimation method 

The student level estimator approach is a modification of the Newton-Raphson iterative method for solving equations 

method outlined by Lord (1980). 

Item Types 

I) Dichotomous items 

With dichotomous items, there must be only one correct answer. This means that the examinee either answers the 

question correctly and gets a full mark or answers the question incorrectly and gets no mark. The common 

dichotomous item types are: 

Multiple choice: this kind of question gives a number of options to choose from (usually four), and the 

examinee has to choose the correct option. 

True–false: this kind of questions presents a statement that is either right or wrong, and the examinee has to 

decide whether it is correct or incorrect. 

Short answer: this kind of question provides a space that the student has to fill in with a short answer. If the 

answer is fixed and no other alternative answer is correct, then the question is dichotomous item. 

II) Polytomous Items 

Items in this category can be partly correct; the examinee might solve part of the question correctly and will receive 

part of the mark. 

Results 

Before estimating question parameters and starting the adaptive test, students were given conventional tests. Students 

answered about 100 questions in three stages. In each stage, they solved an exam of 34 questions. Forty-five students 

of the intermediate class were examined in these conventional tests. These questions were of a first-intermediate 

level. The questions related to three math topics of first-intermediate level. 

The least square method was used to estimate the difficulty level and discrimination parameters of each question. We 

used this method because it decreases the load of using computer processors, and requires no statistical tables such as 

chi-square. 

85

It is agreed that the difficulty level ranges from +3, which is very difficult level, to –3, which is a very easy level 

(Baker, 2001). Table 1 shows that the level of difficulty of the questions ranges from +2.9 to –2.9. This means that 

the questions cover all possible ability levels of students. In table 2, discrimination ranges from –0.3 to –1.22. 

Positive discrimination values means that the question is not valid and should be revised. There are questions with 

discrimination greater than –0.3. However, we do not include them in the process of selection of items because they 

will not be good items because they fall outside the scope of almost all students being tested and would not 

contribute anything of value to the results. 

Table 2. Analysis Of question difficulty 

Max difficulty Min difficulty Average difficulty 

2.9 –2.9 0.51 

Table 3. Analysis of discrimination 

Max discrimination Min discrimination Average discrimination 

–0.3 –3.15 –1.22 

After the conventional test we tested the adaptive assessment on 24 students. Two of the students did not complete 

the exam. According to the hypothesis, we should have modified the formula to shorten the number of questions to 

reach the stable level. We started the adaptive test with the conventional level of the students. Each student started 

with his previous level and with the consideration that we were certain of 85%, that is, σ = 0.15. The graph in Figure 

1 shows the conformance of the adaptive test level and the conventional test level. 

4 

3 

2 

1 

0 

-1 

-2 

-3 

1 3 5 7 9 11 13 15 17 19 21 

Coventional Level Adaptive Level 

Figure 4. Conformance between conventional and adaptive levels 

In Figure 4, about 20% of the points of the conventional levels are not close enough to their corresponding points in 

the adaptive level. While the average student level in the conventional tests is –0.17, the average in the adaptive tests 

is 0.031. Transforming these values to the 100 scale, we find that these values are 47.14 and 50.51, respectively. We 

could say that the averages are almost near each other. The difference between the average of the conventional level 

and the adaptive level is –0.20. In the 100 scale it is –3.37. 

As an example, the fourth point shows that the conventional level is close to the adaptive level: the conventional 

level is about 0.47 and the adaptive level is about 0.2. This means that the difference is 0.27. In the 100 scale, this 

difference is about 4 marks. The nineteenth point shows that the conventional level is 0.13 and the adaptive level is 

1.9; the difference between these points is 1.8. In the 100 scale, the difference is about 29 marks. 

The correlation between the conventional level and the adaptive level is 0.63, which is considered to be highmoderate 

relation. The correlation is considered high if it is 0.70 and above. 

It is not strange to have some deviation in the results of the two kinds of exams due to human error. After elimination 

of 20% of the points that are very far from their corresponding points, we get Figure 5, which is more accurate than 

86

Figure 4. The average difference of the conventional level and the adaptive level in this case 2 is 0.028. In the 100 

scale, it is 0.46. 

After the elimination of 20% of the points, the correlation increases and becomes 0.74. This value is considered to be 

high. 

3 

2 

1 

0 

-1 

-2 

-3 

1 3 5 7 9 11 13 15 

Coventional Level Adaptive Level 


In Figure 6, we started with the subject difficulty as a starting point of the adaptive test. The results here were done 

by simulation of real students’ results. Notice that the previous chart had a student’s initial conventional level as a 

starting point. In this graph, we could say that the conformance of the two lines is acceptable. The students’ 

conventional levels were the same as they were in the previous chart. While the average of these students’ levels in 

conventional tests is –0.17, the average in adaptive tests is –0.0036. Transforming these values to the 100 scale, we 

find that these values are 47.14 and 49.9, respectively. Both charts’ results are almost the same average. The average 

difference of the conventional levels and adaptive levels is –0.12. In the 100 scale it is –2.8. The correlation between 

the conventional and adaptive tests levels is 0.81. The average number of questions to reach the stable or final level 

is 12 questions. 

A question arises after finding that both starting points, the students’ conventional level or the difficulty of the 

subject, reached final levels after an average of 12 questions. Why do the levels of students change? This could be 

interpreted by the fact that students’ levels changed based upon their readiness for the exam, motivation, time of 

exam, health and other conditions, and the fact that some students wanted to get better results. However, those 

changes were within an acceptable range in general. 

Our improved IRT formula and model 

4 

3 

2 

1 

0 

-1 1 

-2 

-3 

3 5 7 9 11 13 15 17 19 21 

Coventional 

Adaptive 


One of the objectives was to find a new formula that contributes new factors to the IRT model. These factors are 

initial student ability, subject’s difficulty, number of exercises covered by the teacher, and number of lessons covered 

by the teacher. As shown in Figure 7, the number of lessons covered and the number of exercises covered directly 

affect the student’s ability level. 

87

Figure 7. Our newly added factors in IRT model 

Moreover, the increase in students’ ability levels will in turn increase ease of subject and vice versa. As a matter of 

fact, the number of exercises and the number of lessons covered do not work by themselves in measuring students’ 

levels because the difficulty of the subject affects these two factors directly. For instance, if the subject is very 

difficult, the increase in number of exercises and lessons might not help in producing good student results. Therefore, 

there should be further research that measures these factors in different subject difficulties. 

For now, we considered these two factors: subject difficulty and initial student ability level. The new factors will 

give the new shape of the formula as: 

θ 

∑ 

i n 

n + 1 = θ n + 

n= 

1 

n 

where (θ0) 

⎧ 

⎪θ 

= init 

θ = 0 ⎨ 

⎪ = 

⎩θ 

diffLevel 

and Iinitial is 

I initial 

⎧ 

⎪ 

= ⎨ 

⎪1 

⎩ 

( 

I 

initial 

n 

s ( θ 

) + 

∑ 

n= 

1 

I 

i 

) 

( θ 

the initail student level from database 

5 ; 

1-Number of 

exercises 

covered 

2-Number of 

lessons 

covered 

the difficulty level of subject if 

if 

if 

θ 

θ 

init 

init 

exists 

does not exist 

n 

) 

increase 

increase 

θ 0 

does not exist 

increases 

4-Subject 

difficulty 

3-Initial 

Student 

ability level 

decreases 

facilitates 

facilitates 

Measuring 

student level 

adaptively 

5 means that the standard error, σ, equals 0.45, because we know that θ0 gives some certainty of the level. In fact, we 

used this value because it is near the middle of the student ability level and will not negatively affect the process of 

estimation. 1 means that the standard error = 1, which means that we are not sure and do not know anything about 

the student level. 

Figure 8 shows the performance of a typical student. It is one of the ideal results that we found. This student started 

in a level of 46 and was almost stable for the first 7 questions. Then his level increased slowly until he became stable 

at 50. The difference is only 4 marks. This change is expected because he might be a little better or a little worse than 

his conventional level. 

Now, consider Figure 9. About 57% of the students follow the best pattern, in which their levels changed little up or 

down. 18 percent of the students were fluctuating in their levels. 15 percent of the students decreased in their levels 

constantly. 10 percent of students increased in their levels constantly. Most students were actually in their correct 

levels from the beginning of the exam. That shows that students actually start almost from the right level position. 

Students whose level changed constantly, increasing or decreasing, were the excellent or weak-level students. 

Students who fluctuated in their level were students who hesitated or revised one or two of the three exam topics. 

88

Figure 8. Number of questions vs. level of student 

abililty Level 

100 

90 

80 

70 

60 

50 

40 

30 

20 

10 

0 

The improved IRT System Implementation 

Ability Level 

70 

65 

60 

55 

50 

45 

40 

35 

30 

25 

20 

15 

10 

5 

0 

1 3 5 7 9 11 13 15 17 19 

No of Question 

1 3 5 7 9 11 13 15 17 19 21 23 25 27 

items- questions 

Figure 9. Number of questions vs. level of a student 

Figure 10. System context diagram 

student Level 

Average 

The model shown in Figure 10 was implemented using C++ as the programming platform and Access as the 

database. The system provides a number of functionalities through a number of processes as illustrated in Figures 11, 

12, 13, and 14. The system facilitates a number of functional requirements such as: a login process; a student 

registration process; add new exams and their questions; create, delete, and edit exam questions; calculate and save 

89

question factors according to IRT; and a wide variety of reports for students’ results. Most importantly, the system is 

the implementation of the adaptive testing process for each student and for a group of students in a given class. 

Admin 

student 

Admin 

Log id & 

password 

Log id 

password 

Deleted class 

or subject info 

New classes 

and subject details 

New user 

info 

Edited user 

info 

Edited classes 

and subject details 

Edited user Info 

Recjecting logon 

1 

Login in 

Activation 

info 

2 

Accounts & 

Basics 

Edited password 

or email info 

Instructor 

Instructor 

Log id & 

password 

D2 

New, deleted or 

Edited Data 

Edited password 

or email info 

Student results 

Activation 

info 

Edited exam & 

question 

Student Info 

Student file 

New exam & 

question 

Deleted exam 

or question infor 

5 

Reporting 

Exam Results 

3 

Question 

Generator 

exam & 

Question & solutions 

Activation 

Info from 

Login subsystem 

Figure 11. The main processes of the system 

Deleted exam 

or question infor 

Edited exam & 

question 

Student 

D1 Exam Storage 

exam & 

Question solutions 

4 

Examine 

Question info 

Solution info 

Exam selection 

The system provides a search based on some criteria, such as exam creation date and the subject of an appropriate 

exam. The student can select the exam he/she wants to write and answer its questions. The system makes sure that 

the factors of IRT exist for each question of that exam. The system selects an appropriate question from the database 

depending on the student’s level and presents the question to the student. Depending on student’s answer, the system 

calculates and saves the level of the student as he/she answers each questions using IRT. If the stopping criteria are 

met, the system finishes the exam. 

In order to achieve all the functional requirements through the system processes, the system deals with a database 

consisting of a number of tables or data stores, including: student table, instructor table, admin table, exam table, 

question table, question choice table, class table, stage table, teach table, subject table, question solution table, and 

student-level table. 

The context diagram shown in Figure 10 gives the empirical overview of the system. There are three main players in 

the system, namely, the instructor, the student, and the system administrator. 

Figure 11 shows the main processes of the system. Process 1 is a main process responsible for auditing the login 

activities of all types of users. Process 2 is responsible for all the user accounts, including new classes, subject 

details, and instructor information. Process 2 is further detailed in Figure 12. Process 3 is the question-generator 

process that stores all questions in the exam storage data store. Process 3 is further detailed in Figure 13. This 

process receives from the instructor the new exam questions, the edited questions, and the deleted questions. Process 

4 is the process for conducting the exams. Process 4 deals with the student and allows exam selection, shows 

question information, and provides question solutions. Process 4 is further detailed in Figure 14. 

90

Updated Stage info 

user-id 

and 

Edited 

info 

2.4 

Edit stage 

2.5 

Edit Subject 

Updated Subject info 

Admin 

admin-id 

password 

User 

2.11 

Edit user 

info 

Reject 

info 

1 

login 

user-id 

and 

Allowed Edited 

info 

edited stage info 

edited Subject info 

Deleted 

Subject 

info 

Updated Class info 

User’s Id and password 

User 

info 

Deleted subject info 

2.6 

Edit Class 

edited Class info 

edited 

user-info 

Email 

& pass 

& logid 

2.13 

Self-edit 

user 

info 

New stage info 

Stage storage 

Subject storage 

2.8 

Delete 

Subject 

User 

file 

modified 

Email 

& pass 

& logid 

user 

info 

Detailed Stage info 

New Subject info 

Detailed Subject info 

New Class Info 

Class storage 

Deleted Stage info 

2.3 

Add Class 

Deleted Class info 

New 

user 

info 

deteted 

info 

user-id 

2.12 

deleted 

info 

Reject 

Delete 

2.2 

Add 

Subject 

Detailed Class info 

Figure 12. Detailed questions process 

2.7 

Delete 

stage 

Class info 

Subject info 

Deleted 

user 

info 

2.1 

Add Stage 

Deleted Stage info 

Deleted Class info 

2.9 

Delete 

Class 

Admin 

2.10 

Add-user 

Admin 

Stage info 

New 

user 

Existing 

user 

91

Instructure 

1 

Login 

User-ID 

Password 

Instructure 

user-ID,password 

User storage 

4.1 

Select 

Exam with 

exam type 

Exam-id 

To conventional 

4.2 

select 

Question 

Exam id 

edited 

Exam info 

Exam id 

test-title 

3.1 

Define 

Exam or 

Assignment 

3.3 

Delete 

Exam or 

assignment 

3.8 

update 

Exam or 

assignment 

info 

Instructure 

Question data 

exam-Id 

Exam- 

Assignment 

info 

Exam 

details info 

Deleted Exam info 

Question 

file 

Exam detailed info 

3.2 

Generate 

Question 

New Question Info 

and answer 

Exam 

details info 

Question 

detailed info 

Updated Exam info 

Exam id 

3.4 

Search 

exam 

Exam info 

Exam info 

Question_id 

For deletion 

3.5 

Selecting 

Question 

3.6 

Delete 

Question 

Deleted quesion 

Question 

file 

question 

Factors(IRT) 

question 

details info 

3.9 

Estimate 

Question 

Factors(IRT) 

Instructure 

Question edited info 

Question_id 

3.7 

editing 

Question 

Updated Question edited info 

solutions 

Solutions 

file 

Figure 13. Detailed question generator process 

Exam-id 

To adaptive 

exist-eaxm-id 

stu-id 

Question 

solution 

stu-id 

Question 

solaition 

end exam 

exam-id 

4.3 

select 

Question 

adaptively 

Question storage 

stu-id 

Termination 

details 

Student 

Level of student 

Question 

details 

Student 

Student 

level 

4.7 

terminate 

end exam 

stu-id 

Question 

solution 

Stu-id & 

student level 

4.4 

Solve 

question 

Solution 

& Question id 

Update 

level 

Student file 

Figure 14. Detailed question solution process 

4.5 

update 

student 

level 

Student 

previvoes 

level 

92

Conclusion 

This paper presented a description of adaptive testing based on IRT and experimented with IRT in order to evaluate 

its applicability and benefits. It also presented enhancements for IRT. Our estimation of IRT questions’ parameters 

was based on the least square method, a well-known statistical method. We demonstrated that it is possible to reduce 

the number of questions in the adaptive test to reach the final level of the students by modifying the equation for the 

estimation of the student’s ability level. In order to make IRT more realistic and applicable, we incorporated new 

factors into IRT, namely initial student ability, subject difficulty, number of exercises covered by the teacher, and 

number of lessons covered by the teacher. Students’ attitudes toward adaptive tests and their results were measured 

by a questionnaire. Our conventional exam results were compared with adaptive results and used in determining IRT 

parameters. We have presented enhancement and modification of the IRT formula of estimating student ability level 

and had positive results in minimizing the number of questions in adaptive tests. 

References 

Al-A'ali, M. (2007). A method for improving adaptive testing by evaluating and improving the item response theory. 

WSEAS Transactions on Information Science and Applications 4 (3), 466-471. 

Baker, F. (2001). The basics of item response theory: ERIC Clearinghouse on assessment and evaluation, College 

Park, MD: University of Maryland. 

Bennett, R. E., & Persky, H. (2002). Problem solving in technology-rich environments. In C. Richardson (Ed.) 

Qualifications and Curriculum Authority: Assessing gifted and talented children, London, England: Qualifications 

and Curriculum Authority, 19–33. 

Eggen, T. J. H. M., & Straetmans, G. J. J. M. (2000). Computerized adaptive testing for classifying examinees into 

three categories. Educational and Psychological Measurement, 60, 713–734. 

Fraley, R. C., Waller, N. G., & Brennan, K. A. (2000). An item response theory analysis of self-report measures of 

adult attachment. Journal of Personality and Social Psychology, 78 (2) 350–365. 

Gouli, E., Kornilakis, H., Papanikolaou, K., & Grigoriado, M. (2001). Adaptive assessment improving interaction in 

an educational hypermedia system, Greece: University of Athens. 

Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory, Newbury Park, 

CA: Sage Publications. 

Hashway, R. M. (1998). Error-free mental measurements: Applying qualitative item response theory to assessment 

and program validation including a developmental theory of assessment, San Francisco, CA: Austin & Winfield. 

Hildegarde, S., & Jacobson, Z. (1997). A comparison of early childhood assessments and a standardized measure for 

program evaluation, Ph.D. Dissertation, Virginia Polytechnic & State University, Blacksburg, VA. 

Linacre, J. M. (2000). Computer-adaptive testing: A methodology whose time has come (MESA Memorandum No. 

69), MESA Psychometric Laboratory, University of Chicago. 

Lord, F. M. (1980). Application of item response theory to practical testing problems, Hillsdale, NJ: Lawrence 

Erlbaum Associates. 

Rudner, L. M. (2002). An examination of decision-theory adaptive testing procedures. Paper presented at the annual 

meeting of the American Educational Research Association, April 1-5, 2002, New Orleans, USA. 

Segall, D. O. (2000). Principles of multidimensional adaptive testing. In W. J. van der Linden & C. A. W. Glas 

(Eds.), Computerized adaptive testing: Theory and practice, Dordrecht: Kluwer Academic, 53-73. 

93

Shute, V., & Towle, B. (2003). Adaptive e-learning, Hillsdale, NJ: Lawrence Erlbaum Associates. 

Stage, C. (1997a). The applicability of item response models to the SweSAT: A study of the DTM subtest, retrieved 

October 15, 2007, from http://www.umu.se/edmeas/publikationer/pdf/enr2197sec.pdf. 

Stage, C. (1997b). The applicability of item response models to the SweSAT: A study of the ERC subtest, retrieved 


Stage, C. (1997c). The applicability of item response models to the SweSAT: A study of the READ subtest, retrieved 


Stage, C. (1997d). The applicability of item response models to the SweSAT: A study of the WORD subtest, retrieved 


Weiss, D. J. (1983). New horizons in testing: Latent trait test theory and computerized adaptive testing, New York, 

NY: Academic Press. 

Wu, I.-L. (2000). Model management system for IRT-based test construction decision support system. Decision 

Support Systems, (27) 4, 443–458, Netherlands: Elsevier. 

Young, J. W. (1991). Gender bias in predicting college academic performance: A new approach using item response 

theory. Journal of Educational Measurement, 28 (1), 37-47. 

94

Chang, S.-H., Lin, P.-C., & Lin, Z. C. (2007). Measures of Partial Knowledge and Unexpected Responses in Multiple-Choice 

Tests. Educational Technology & Society, 10 (4), 95-109. 

Measures of Partial Knowledge and Unexpected Responses in Multiple-Choice 

Tests 

Shao-Hua Chang 

Department of Applied English, Southern Taiwan University, Tainan, Taiwan // shaohua@mail.stut.edu.tw 

Pei-Chun Lin 

Department of Transportation and Communication Management Science, National Cheng Kung University, Taiwan 

peichunl@mail.ncku.edu.tw 

Zih-Chuan Lin 

Department of Information Management, National Kaohsiung First University of Science & Technology, Taiwan 

u9324819@ccms.nkfust.edu.tw 

ABSTRACT 

This study investigates differences in the partial scoring performance of examinees in elimination testing and 

conventional dichotomous scoring of multiple-choice tests implemented on a computer-based system. 

Elimination testing that uses the same set of multiple-choice items rewards examinees with partial knowledge 

over those who are simply guessing. This study provides a computer-based test and item analysis system to 

reduce the difficulty of grading and item analysis following elimination tests. The Rasch model, based on item 

response theory for dichotomous scoring, and the partial credit model, based on graded item response for 

elimination testing, are the kernel of the test-diagnosis subsystem to estimate examinee ability and itemdifficulty 

parameters. This study draws the following conclusions: (1) examinees taking computer-based tests 

(CBTs) have the same performance as those taking paper-and-pencil tests (PPTs); (2) conventional scoring does 

not measure the same knowledge as partial scoring; (3) the partial scoring of multiple choice lowers the number 

of unexpected responses from examinees; and (4) the different question topics and types do not influence the 

performance of examinees in either PPTs or CBTs. 

Keywords 

Computer-based tests, Elimination testing, Unexpected responses, Partial knowledge, Item response theory 

Introduction 

The main missions of educators are determining learning progress and diagnosing difficulty experienced by students 

when studying. Testing is a conventional means of evaluating students, and testing scores can be adopted to observe 

learning outcomes. Multiple-choice (MC) items continue to dominate educational testing owing to their ability to 

effectively and simply measure constructs such as ability and achievement. Measurement experts and testing 

organizations prefer the MC format to others (e.g., short-answer, essay, constructed-response) for the following 

reasons: 

Content sampling is generally superior to other formats, and the application of MC formats normally leads to 

highly content-valid test-score interpretations. 

Test scores can be extremely reliable with a sufficient number of high-quality MC items. 

MC items can be easily pre-tested, stored, used, and reused, particularly with the advent of low-cost, 

computerized item-banking systems. 

Objective, high-speed test scoring is achievable. 

Diagnostic subscores are easily obtainable. 

Test theories (i.e., item response, generalizability, and classical) easily accommodate binary responses. 

Most content can be tested using this format, including many types of higher-level thinking (Haladyna & 

Downing, 1989). 

However, the conventional MC examination scheme requires examinees to evaluate each option and select one 

answer. Examinees are often absolutely certain that some of the options are incorrect, but still unable to identify the 

correct response (Bradbard, Parker, & Stone, 2004). From the viewpoint of learning, knowledge is accumulated 

continuously rather than on an all-or-nothing basis. The conventional scoring format of the MC examination cannot 






95

distinguish between partial knowledge (Coombs, Milholland, & Womer, 1956) and the absence of knowledge. In 

conventional MC tests, students choose only one response. The number of correctly answered questions is counted, 

and the scoring method is called number scoring (NS). Akeroyd (1982) stated that NS makes the simplifying 

assumption that all of the wrong answers of students are the results of random guesses, thus neglecting the existence 

of partial knowledge. Coombs et al. (1956) first proposed an alternative method for administering MC tests. In their 

procedure, students are instructed to mark as many incorrect options as they can identify. This procedure is referred 

to as elimination testing (ET). Bush (2001) presented a multiple-choice test format that permits an examinee who is 

uncertain of the correct answer to a question to select more than one answer. Incorrect selections are penalized by 

negative marking. The aim of both the Bush and Coombs schemes is to reward examinees with partial knowledge 

over those who are simply guessing. 

Education researchers have been continuously concerned not only about how to evaluate students’ partial knowledge 

accurately but also about how to reduce the number of unexpected responses. The number of correctly answered 

questions is composed of two numbers: the number of questions to which the students actually know the answer, and 

the number of questions to which the students correctly guess the answer (Bradbard et al., 2004). A higher frequency 

of the second case indicates a less reliable learning performance evaluation. Chan & Kennedy (2002) compared 

student scores on MC and equivalent constructed-response questions, and found that students do indeed score better 

on constructed-response questions for particular MC questions. Although constructed-response testing produces 

fewer unexpected responses than the conventional dichotomous scoring method, the change of item constructs raises 

the complexity of both creating the test and of the post-test item grading and analysis, whereas ET uses the same set 

of MC items and makes guessing a futile effort. 

Bradbard et al. (2004) suggested that the greatest obstacle in implementing ET is the complexity of grading and the 

analysis of test items following traditional paper assessment. Accordingly, examiners are not very willing to adopt 

ET. To overcome this problem, this study provides an integrated computer-based test and item-analysis system to 

reduce the difficulty of grading and item analysis following testing. Computer-based tests (CBTs) offer several 

advantages over traditional paper-and-pencil tests (PPTs). The benefits of CBTs include reduced costs of data entry, 

improved rate of disclosure, ease of data conversion into databases, and reduced likelihood of missing data (Hagler, 

Norman, Radick, Calfas, & Sallis, 2005). Once set up, CBTs are easier to administer than PPTs. CBTs offer the 

possibility of instant grading and automatic tracking and averaging of grades. In addition, they are easier to 

manipulate to reduce cheating (Inouye & Bunderson, 1986; Bodmann & Robinson, 2004). 

Most CBTs measure test item difficulty based on the percentage of correct responses. A higher percentage of correct 

responses implies an easier test item. The approach of test items analysis disregards the relationship between the 

examinee’s ability and item difficulty. For instance, if the percentage of correct responses for test item A is quite 

small, then the test item analysis system categorizes it as “difficult.” However, statistics also reveal that more failing 

examinees than passing examinees answer item A correctly. Therefore, the design of test item A may be 

inappropriate, misleading, or unclear, and should be further studied to aid future curriculum designers to compose 

high-quality items. To avoid the fallacy of percentage of correct responses, this study constructs a CBT system that 

applies the Rasch model based on item response theory for dichotomous scoring and the partial credit model based 

on graded item response for ET to estimate the examinee ability and item difficulty parameters (Baker, 1992; 

Hambleton & Swaminathan, 1985; Zhu & Cole, 1996; Wright & Stone, 1979; Zhu, 1996; Wright & Masters, 1982). 

Before ET implemented by computer-based system is broadly adopted, we still need to examine whether any 

discrepancy exists in the performance of examinees who take elimination tests on paper and the performance of those 

who take CBTs. This study compares the scores of examinees taking tests using the NS of dichotomous scoring 

method and the partial scoring of ET using the same set of MC items in CBT and PPT settings, where the content 

subject is operations management. This study has the following specific goals: 

1. Evaluate whether the partial scoring for the MC test produces fewer unexpected responses of examinees. 

2. Compare the examinee performance on conventional PPTs with their performance on CBTs. 

3. Analyze whether different question content, such as calculation and concept, influences the performance of 

examinees on PPTs and CBTs. 

4. Investigate the relationship between an examinee’s ability and the item difficulty, to help the curriculum 

designers compose high-quality items. 

96

The rest of this paper will first present a brief literature review on partial knowledge, testing methods, scoring 

methods, multiple choice, and CBTs. Then this paper will describe the configuration of a computer-based assessment 

system, formulate the research hypotheses, and provide the research method, experimental design, and data 

collection in detail. The statistics analysis and hypothesis testing results will be presented subsequently. Conclusions 

are finally drawn in the last section. 

Related literature 

This study first defines related domain knowledge, discusses studies applied to conventional scoring and partial 

scoring, compares scoring modes and investigates the principle of designing MC items, and finally summarizes the 

pros and cons of CBT systems. 

Partial knowledge 

Reducing the opportunities to guess and measuring partial knowledge improve the psychometric properties of a test. 

These methods can be classified by their ability to identify partial knowledge on a given test item (Alexander, 

Bartlett, Truell, & Ouwenga, 2001). Coombs et al. (1956) stated that the conventional scoring format of the MC 

examination cannot distinguish between partial knowledge and absence of knowledge. Ben-Simon, Budescu, & 

Nevo (1997) classify examinees’ knowledge for a given item as full knowledge (identifies all of the incorrect 

options), partial knowledge (identifies some of the incorrect options), partial misinformation (identifies the correct 

answer and some incorrect options), full misinformation (identifies only the correct answer), and absence of 

knowledge (either omits the item or identifies all options). Bush (2001) conducted a study that allows examinees to 

select more than one answer to a question if they are uncertain of the correct one. Negative marking is used to 

penalize incorrect selections. The aim is to explicitly reward examinees who possess partial knowledge as compared 

with those who are simply guessing. 

Number-scoring (NS) of multiple choice 

Students choose only one response. The number of correctly answered questions is composed of the number of 

questions to which the student knows the answer, and the number of questions to which the students correctly guess 

the answer. According to the classification of Ben-Simon et al. (1997), NS can only distinguish between full 

knowledge and absence of knowledge. A student’s score on an NS section with 25 MC questions and three points per 

correct response is in the range 0–75. 

Elimination testing (ET) of multiple choice 

Alternative schemes proposed for administering MC tests increase the complexity of responding and scoring, and the 

available information about student understanding of material (Coombs et al., 1956; Abu-Sayf, 1979; Alexander et 

al., 2001). Since partial knowledge is not captured in conventional NS format of an MC examination, Coombs et al. 

(1956) describe a procedure that instructs students to mark as many incorrect options as they can identify. One point 

is awarded for each incorrect choice identified, but k points are deducted (where k equals the number of options 

minus one) if the correct option is identified as incorrect. Consequently, a question score is in the range (–3, +3) on a 

question with four options, and a student’s score on an ET section with 25 MC questions with four options each is in 

the range (–75, +75). Bradbard and Green (1986) have classified ET scoring as follows: completely correct score 

(+3), partially correct score (+2 or +1), no-understanding score (0), partially incorrect score (–1 or –2), completely 

incorrect score (–3). 

Subset selection testing (SST) of multiple choice 

Rather than identifying incorrect options, the examinee attempts to construct subsets of item options that include the 

correct answer (Jaradat & Sawaged, 1986). The scoring for an item with four options is as follows: if the correct 

97

esponse is identified, then the score is 3; while if the subset of options identified includes the correct response and 

other options, then the item score is 3 – n (n = 1, 2, or 3), where n denotes the number of other options included. If 

subsets of options that do not include the correct option are identified, then the score is –n, where n is the number of 

options included. SST and ET are probabilistically equivalent. 

Comparison studies of scoring methods 

Coombs et al. (1956) observed that tests using ET are somewhat more reliable than NS tests and measure the same 

abilities as NS scoring. Dressel & Schmid (1953) compared SST with NS using college students in a physical science 

course. They observed that the reliability of the SST test, at 0.67, was slightly lower than that of the NS test at 0.70. 

They also noted that academically high-performing students scored better than average compared to low-performing 

students, with respect to full knowledge, regardless of the difficulty of the items. Jaradat and Tollefson (1988) 

compared ET with SST, using graduate students enrolled in an educational measurement course. No significant 

differences in terms of reliability were observed between the methods. Jaradat and Tollefson reported that the 

majority of students felt that ET and SST were better measures of their knowledge than conventional NS, but they 

still preferred NS. Bradbard et al. (2004) concluded that ET scoring is useful whenever there is concern about 

improving the accuracy of measuring a student’s partial knowledge. ET scoring may be particularly helpful in 

content areas where partial or full misinformation can have life-threatening consequences. This study adopted ET 

scoring as the measurement scheme for partial scoring. 

Design of multiple choice 

An MC item is composed of a correct answer and several distractors. The design of distractors is the largest 

challenge in constructing an MC item (Haladyna & Downing, 1989). Haladyna & Downing summarized the common 

rules of design found in many references. One such rule is that all the option choices should adopt parallel grammar 

to avoid giving clues to the correct answer. The option choices should address the same content, and the distractors 

should all be reasonable choices for a student with limited or incorrect information. Items should be as clear and 

concise as possible, both to ensure that students know what is being asked, and to minimize reading time and the 

influence of reading skills on performance. Haladyna and Downing recommended some guidelines for developing 

distractors: 

Employ plausible distractors; avoid illogical distractors. 

Incorporate common student errors into distractors. 

Adopt familiar yet incorrect phrases as distractors. 

Use true statements that do not correctly answer the items. 

Kehoe (1995) recommended improving tests by maintaining and developing a pool of “good” items from which 

future tests are drawn in part or in whole. This approach is particularly true for instructors who teach the same course 

more than once. The proportion of students answering an item correctly also affects its discrimination power. Items 

answered correctly (or incorrectly) by a large proportion of examinees (more than 85%) have a markedly low power 

to discriminate. In a good test, most items are answered correctly by 30% to 80% of the examinees. Kehoe described 

the following three methods to enhance the ability of items to discriminate among abilities: 

Items that correlate less than 0.15 with total test score should probably be restructured. 

Distractors that are not chosen by any examinees should be replaced or eliminated. 

Items that virtually all examinees answer correctly are unhelpful for discriminating among students and should 

be replaced by harder items. 

Pros and cons of CBTs 

CBTs have several benefits: 

The single-item presentation is not restricted to text, is easy to read, and allows combining with pictures, voice, 

image, and animation. 

98

CBTs shorten testing time, give instantaneous results, and increase test security. The online testing format of an 

exam can be configured to enable instantaneous feedback to the student, and can be more easily scheduled and 

administered than PPTs (Gretes & Green, 2000; Bugbee, 1996). 

The test requires no paper, eliminating the battle for the copy machine. The option of printing the test is always 

available. 

The student can take the test when and where appropriate. Even taking the test at home is an option if the 

student has Internet access at home. Students can wait until they think they have mastered the material before 

being tested on it. 

The test is a valuable teaching tool. CBTs provide immediate feedback, requiring the student to get the correct 

answer before moving on. 

CBTs save 100% of the time taken to distribute the test to test-takers (a CBT never has to be handed out). 

CBTs save 100% of the time taken to create different versions of the same test (re-sequencing questions to 

prevent an examinee from cheating by looking at the test of the person next to him). 

However, CBTs have some disadvantages: 

The test format is normally limited to true/false and MC. Computer-based automatic grading cannot easily 

judge the accuracy of constructed-response questions such as short-answer, problem-solving exercises, and 

essay questions. 

When holding an onsite test, instructors must prepare many computers for examinees, and be prepared for the 

difficulties caused by computer crashes. 

The computer display is not suitable for question items composed of numerous words, since the resolution 

might make the text difficult to read (Mazzeo & Harvey, 1988). 

Most items in mathematics and chemistry testing need manual calculation. The need to write down and 

calculate answers on draft paper might decrease the answering speed (Ager, 1993). 

Research method 

This study first constructs a computer-based assessment system, and then adopts experimental design to record an 

examinee’s performance with different testing tools and scoring approaches. This section describes research 

hypotheses, variables, the experimental design, and data collection. 

System configuration 

As well as providing a platform for computer-based ET, this system implements the Rasch one-parameter logistics 

item characteristics curve (ICC) model for dichotomous scoring and the grade-response model for partial scoring of 

ET to estimate the item and ability parameters (Hambleton & Swaminathan, 1985; Wright & Masters, 1982; Wright 

& Stone, 1979; Zhu, 1996; Zhu & Cole, 1996). Furthermore, item difficulty analysis allows the system to maintain a 

high-quality test bank. shows the system configuration, which comprises databases and three main subsystems: (1) 

The computer-based assessment system is the platform used by examinees to take tests, and enables learning 

progress to be tracked, grades to be queried, and examinees’ response patterns to be recorded in detail; (2) the 

testbank management system, the platform used by instructors to manage testing items and examinee accounts; (3) 

the test diagnosis system, which collects data from the answer record and the gradebook database to analyze the 

difficulty of test items and the ability of examinees. These subsystems are described in detail as follows (Figure 1): 

Computer-based assessment system 

The computer-based assessment system links to the answer record, and the gradebook database collects the complete 

answer record and incorporates a feedback mechanism that enables examinees to check their own learning progress 

and increase their learning efficiency by means of constructive interaction. The main functions of the computerbased 

assessment system are as follows: The system first verifies the examinee’s eligibility for the test and then 

displays the test information, including allowed answering time, scoring methods, and test items. The system permits 

examinees to write down keywords or mark on the test sheet, as in a paper test. The complete answering processes 

99

are stored in the answer record and the gradebook database. The answer record and gradebook database enable the 

computer-based assessment system to collect information for examinees during tests, improving examinees’ 

understanding of their own learning status. Examinees can understand the core of a problem by reading the remarks 

they themselves made during a test to identify any errors and improve in areas of poor understanding. 

Instructor 

interface 

Testbank management system 

Test item management/ 

Exams management/ 

Account management 

Testbank 

Database 

Examinee 

interface 

Computer-based assessment system 

Online testing / 

Progress tracking/ 

Grade querying 

Answer record 

and Gradebook 

Database 

Test Diagnosis system 

Defining difficulty 

Setting up commonly used tests 

Testing unexpected responses 

Item fit analysis Person fit analysis 

Estimating ability and item parameter 

Scoring 

Sorting examinee’s responses 

Figure 1. System configuration 

100

Testbank management system 

The testbank management system is normally accessed by instructors who are designing test items, revising test 

items, designing examinations, and reusing tests. The main advantage of the item banking is in test development. 

This system allows a curriculum designer to edit multiple-choice questions, constructed-respond items, and true/false 

items and specify scoring modes. Designed test items are stored in the testbank database. Test items can be displayed 

in text format or integrated with multimedia images. This system supports three parts of the preparation of 

examinations: (1) creating original questions, (2) browsing questions from a testbank, and (3) using questions from a 

testbank by selecting questions items at random. The system provides dichotomous scoring and partial scoring for 

MC items. After the item sheet has been prepared, the curriculum designer must specify “examinee eligibility,” 

“testing date,” “scoring mode,” “time allowed,” and “question values.” The computer-based assessment system 

permits only eligible examinees to take the test at the specified time. Although the system can automatically and 

instantly grade multiple-choice and true/false questions, the announcement of testing scores is delayed until all 

examinees have finished the test. 

Test diagnosis system 

The test diagnosis system analyzes items and examinees’ ability based on scoring methods (dichotomous scoring and 

partial scoring) and the data retrieved from the answer record and gradebook database. The diagnostic process is 

summarized as follows: (1) Sorting the results from the test records. The system first sorts the item-response patterns 

for all examinees. The answer records are rearranged according to each examinee’s ability and the number of 

students who have given the correct answer to a given question. (2) Matrix reduction. The system then deletes 

useless item responses and examinee responses, such as those questions to which all examinees gave correct answers 

or wrong answers, or received a full or a zero score, since these data do not help to assess the ability of examinees 

(Baker, 1992; Bond & Fox, 2001). Matrix reduction can reduce the resources and time required to conduct the 

calculation. (3) The JMLE procedures (Hambleton & Swaminathan, 1985; Wongwiwatthananukit, Popovich, and 

Bennett, 2000) depicted in Figure 2 are applied to estimate the ability and item parameters. Then the diagnosis 

procedures are followed by the item and person fit analysis, and the procedures are described as follows: First, let 

random variable X ni denote examinee n’s response on item i, in which X ni = 1 signifies the correct answer; θ n is 

the personal parameter of examinee n; and b i denotes the item parameter, which determines the item location and is 

called the item difficulty in attainment tests. The expected value and variance of X ni are shown in equations (1) and 

(2). 

(1) EX ( ni) = exp( θn − bi) /[1+ exp( θn − bi)] 

= πni 

(2) Var( X ) = π (1 − π ) = W 

ni ni ni ni 

The standardized residual and kurtosis of X ni are shown in equations (3) and (4). 

(3) Zni = [ Xni − E( Xni )]/[ Var( Xni 

)] 

(4) C 

4 4 

= [(1 − π ) ( π )] + [(0 −π ) (1 − π )] 

ni ni ni ni ni 

The person fit analysis shows the mean square (MNSQ) and standardized weighted mean square (Zstd), as 

represented in equations (5) and (6). 

1 2 

2 

(5) MNSQ = ∑WniZni ∑ Wni = ν n 

i i 

13 

(6) Zstd = tn = ( ν n − 1(3/ ) qn) + ( qn 

/3) where = ∑( − ) ∑ 

q C W ( W ) 

2 2 2 

n ni ni ni 

i i 

The item fit analysis shows the mean square (MNSQ) and standardized weighted mean square (Zstd) are as follows: 


2 

(7) MNSQ = ∑WniZ ni ∑ Wni = ν i 

n n 

13 

(8) Zstd = ti = ( ν i − 1(3/ ) qi) + ( qi 

/3) where = ∑( − ) ∑ 

q C W ( W ) 

2 2 2 

i ni ni ni 

n n 

Figure 2. JMLE procedures for estimating both ability and item parameters 

Table 1 lists the principles to distinguish examinees and items from unexpected responses, summarized by Bond and 

Fox (2001), and Linacre and Wright (1994). The acceptable range for MNSQ is between 0.75 and 1.3, and the Zstd 

value should be in the range (-2, +2). Person and item fitness outside the range are considered unexpected responses. 

Incorporating the calculation module in the online testing system is helpful for curriculum designers to maintain the 

quality of test items by modifying or removing test items with unexpected responses. 

Table 1. Fit statistics 

MNSQ Zstd Variation Misfit Type 

>1.3 >2.0 Too much Underfit 

Research hypotheses 

This study proposes four hypotheses based on literature reviews. First, Alexander et al. (2001) explored students in a 

computer technology course who completed either a PPT or a CBT in a proctored computer lab. The test scores were 

similar, but students in the computer-based group, particularly freshmen, completed the test in the least amount of 

time. Bodman and Robinson (2004) investigated the effect of several different modes of test administration on scores 

and completion times, and the results of the study indicate that undergraduates completed the computer-based tests 

faster than the paper-based tests with no difference in scores. Stated formally: 

Hypothesis 1. The average score of CBTs is equivalent to that of PPTs. 

Second, the statistical analysis of Bradbard et al. (2004) indicates that the Coombs procedure is a viable alternative to 

the standard scoring procedure. In Ben-Simon et al.’s (1997) classification, NS performed very poorly in 

discriminating between full knowledge and absence of knowledge. Stated formally: 

Hypothesis 2. ET detects partial knowledge of examinees more effectively than NS. 

Third, Bradbard and Green (1986) indicated that elimination testing lowers the amount of guesswork, and the 

influence increases throughout the grading period. Stated formally: 

Hypothesis 3. ET lowers the number of unexpected responses for examinees more effectively than NS. 

Finally, most items in mathematics and chemistry testing need manual calculation. The need to write down and 

calculate answers on draft paper might lower the answering speed (Ager, 1993). Stated formally: 

Hypothesis 4. Different types of question content, such as calculation and concept, influence the performance of 

examinees on PPTs or CBTs. 

Research variables 

The independent variables used in this study and the operational definitions are as follows: 

Scoring mode: including partial scoring and conventional dichotomous scoring to analyze the influence of 

different answering and scoring schemes on partial knowledge. 

Testing tool: comparing conventional PPTs with CBTs and recognizing appropriate question types for CBTs. 

The dependent variable adopted in this study is the students’ performance, which is determined by test scores. 

Experimental design 

Tests were provided according to the two scoring modes and two testing tools, which were combined to form four 

treatments. Table 2 lists the multifactor design. Treatment 1 (T1) was CBTs, using the ET scoring method; 

Treatment 2 (T2) was CBTs, using NS scoring method; Treatment 3 (T3) was PPTs, using ET scoring method; and 

Treatment 4 (T4) was PPTs, using NS scoring method. 

Data collection 

Table 2. Multifactor design 

CBTs PPTs 

ET T1 T3 

NS T2 T4 

The subjects of the experiment were 102 students in an introductory operations management module, which is a 

required course for students of the two junior classes in the Department of Information Management at National 

Kaohsiung First University of Science and Technology in Taiwan. All students were required to take all four exams 


in the course. A randomized block design, separating each class into two cohorts, was adopted for data collection 

before the first test. The students were thus separated into the following four groups: class A, cohort 1 (A1); class A, 

cohort 2 (A2); class B, cohort 1 (B1); and class B, cohort 2 (B2). The four tests were implemented separately using 

CBTs and PPTs. Each test was worth 25% of the final grade. The item contents were concept oriented and 

calculation oriented. Because all subjects participating in this study needed to take both NS and ET scored tests, they 

were given a lecture describing ET and given several opportunities to practice, to prevent bias in personal scores due 

to unfamiliarity with the answering method, thus enhancing the reliability of this study. Students in each group were 

all given exactly the same questions. 

Data analysis 

Reliability 

The split-half reliability coefficient was calculated to ensure internal consistency within a single test for each group. 

Linn and Gronlund (2000) proposed setting the reliability coefficient between 0.60 and 0.85. Table 3 lists the 

reliability coefficient for each treatment and cohort combination, and all Cronbach α values were in this range. 

Table 3. Reliability coefficients for each group 

Cronbach α 

Test 1 .6858 (T2, A1) .6684 (T3, A2) .6307 (T4, B1) .6901 (T1, B2) 

Test 2 .6649 (T4, A1) .7011 (T1, A2) .6053 (T2, B1) .7791 (T3, B2) 

Test 3 .7450 (T4, A2) .7174 (T1, A1) .7043 (T2, B2) .8078 (T3, B1) 

Test 4 .7358 (T2, A2) .7270 (T3, A1) .7041 (T4, B2) .6341 (T1, B1) 

Correlation analysis 

The Pearson and Spearman correlation coefficients were derived to determine whether response patterns of the 

examinees were consistent. Table 4 lists the correlation coefficient of ET and NS on each test, and Table 5 presents 

the correlation coefficients of CBTs and PPTs. The analytical results in both tables demonstrate a highly positive 

correlation for each scoring mode and test tool. 

Table 4. Correlation coefficient of ET vs. NS 

Pearson Spearman 

Test 1 

CBTs 

0.894 

PPTs CBTs PPTs 

** 

0.903 ** 

0.887 ** 

0.902 ** 

Test 2 0.698 ** 

Test 3 0.747 ** 

0.692 ** 

0.787 ** 

0.679 ** 

0.676 ** 

0.722 ** 

0.833 ** 

Test 4 0.871 ** 

0.768 ** 

0.887 ** 

**indicates significance at the 0.01 level for two-tailed test. 

0.743 ** 

Table 5. Correlation coefficient of CBTs vs. PPTs 

Pearson Spearman 

NS ET NS ET 

Test 1 .868 ** 

.836 ** 

.875 ** 

Test 2 .759 ** 

.846 ** 

.791 ** 

Test 3 .840 ** 

.805 ** 

.802 ** 

Test 4 .832 ** 

.830 ** 

.842 ** 

**indicates significance at the 0.01 level for two-tailed test. 

.800 ** 

.803 ** 

.766 ** 

.796 ** 


Hypothesis testing 

One-way analysis of variation (ANOVA) was adopted to test Hypothesis 1. The average score of the CBTs is 

equivalent to that of PPTs. Table 6 shows that all p-values are greater than 0.05. Each individual group is not 

statistically significant at the 5% level and no sufficient proof is available to reject Hypothesis 1. 

Test 1 

Test 2 

Test 3 

Test 4 

Table 6. One-way ANOVA 

Scoring 

mode 

Mean of score F-statistic p-value 

NS 

45.60 (PPTs) 

42.84 (CBTs) 

1.622 .209 

ET 

NS 

ET 

NS 

ET 

NS 

ET 

27.16 (PPTs) 

32.08 (CBTs) 

55.12 (PPTs) 

55.44 (CBTs) 

43.32 (PPTs) 

39.72 (CBTs) 

47.04 (PPTs) 

49.44 (CBTs) 

44.12 (PPTs) 

45.70 (CBTs) 

41.12 (PPTs) 

36.48 (CBTs) 

30.72 (PPTs) 

24.40 (CBTs) 

1.509 .225 

0.16 .901 

.671 .417 

.495 .485 

.143 .707 

1.688 .200 

2.223 .143 

Hypothesis 2: ET can detect partial knowledge of examinees more effectively than NS. According to the scoring 

classification of Bradbard and Green (1986), and Ben-Simon et al.’s classification (1997), Table 7 summarizes the 

average number of items for NS and ET scoring taxonomy. For Test 1, the average number of correct answers for 

class A, cohort 1 was 14.28; the number of answers revealing full knowledge for class A, cohort 2 was 11.67; the 

number of answers indicating partial knowledge for class A, cohort 2 was 2.38; the number of answers revealing 

absence of knowledge for class A, cohort 2 was 1.42; the number of answers indicating partial misinformation for 

class A, cohort 2 was 9.25; and the number of answers revealing full misinformation for class A, cohort 2 was 0.29. 

Test 1 

Test 2 

Test 3 

Test 4 

Table 7. Average number of items for scoring taxonomy 

NS ET 

correct full knowledge 

partial 

knowledge 

absence of 

knowledge 

partial 

misinformation 

full 

misinformation 

14.28 (A1) 11.67 (A2) 2.38 1.42 9.25 0.29 

15.20 (B1) 12.48 (B2) 2.56 1.36 8.44 0.16 

18.38 (A1) 14.16 (A2) 2.40 2.20 6.20 0.04 

18.48 (B1) 15.12 (B2) 2.40 2.24 5.08 0.16 

15.88 (A2) 15.33 (A1) 3.46 1.13 4.83 0.25 

16.48 (B2) 15.80 (B1) 2.12 0.96 6.08 0.04 

12.16 (A2) 11.08 (A1) 3.63 2.83 7.38 0.08 

13.16 (B2) 10.36 (B1) 2.56 2.68 9.08 0.32 


To test whether ET can effectively detect partial knowledge of examinees, the number of correct items of NS was 

compared to the number of full knowledge of ET in each test. Examinees who did not know the correct answer to an 

NS test item would have either guessed or given up answering. The number of correctly answered items is composed 

of (1) correct response by lucky blind guesses, and (2) correct response by examinees’ knowledge. Burton (2002) 

proposed that the conventional MC scoring can be described by equation B = (K + k + R), where B is the number of 

correct items; K is the number of correct items for which an examinee possesses accurate knowledge; k is the number 

of correct items for which an examinee possesses partial knowledge and guesses correctly; R is the number of correct 

items for which the examinee has no knowledge but makes a lucky blind guess. Burton considered that the examinee 

could delete distractors and increase the proportion of correct guesses based on partial knowledge. Our study 

randomly assigns subjects to each group so the ability should be approximately equivalent. Table 7 demonstrates that 

the number of correct NS items for each of the four tests is greater than the full knowledge number of ET, which 

shows that ET can distinguish between full knowledge and partial knowledge by partial scoring. 

Hypothesis 3: ET lowers the number of unexpected responses for examinees more effectively than NS. Table 8 

shows the number of unexpected responses based on the calculation described in the research method section, 

equations (1) to (8), and reveals that the number of unexpected responses in NS is greater than ET for each test, 

which demonstrates that ET reduces the unexpected responses of examinee. 

Table 8. Number of unexpected responses 

NS ET 

Test 1 25 17 

Test 2 10 9 

Test 3 17 7 

Test 4 29 15 

Next, one-way ANOVA was adopted to test Hypothesis 4, that different question contents, such as calculation and 

concept, influence the performance of examinees on PPTs or CBTs. Table 9 shows that p-value > 0.05 for each 

subgroup. Thus, experimental results reject Hypothesis 4, and indicate that different question content does not 

influence the performance of examinees on PPTs or CBTs. This result did not confirm Ager’s study (1993). The first 

reason might be the course content: an introductory course focused on principles and concepts of operations 

management, instead of lots of complex calculation. The second explanation is the subjects in this study are all 

students from the information management department. These students are quite used to interacting with computer 

interfaces; consequently, the performance difference between PPTs and CBTs is insignificant. 

Test 1 

PPTs 

CBTs 

Test 2 PPTs 

Table 9. One-way ANOVA 

Scoring Mean F-statistic p-value 

NS 

ET 

NS 

ET 

NS 

ET 

1.97 (concept) 

1.87 (calculation) 




.90 (calculation) 

.385 (concept) 






.163 .688 

.744 .394 

.206 .666 

.000 .989 

.014 .907 

.552 .462 


Conclusion 

Test 3 

Test 4 

CBTs 

PPTs 

CBTs 

PPTs 

CBTs 

NS 

ET 

NS 

ET 

NS 

ET 

NS 

ET 

NS 

ET 





















.304 .591 

.007 .936 

.054 .818 

.072 .789 

.005 .944 

.052 .831 

.248 .622 

3.222 084 

1.229 .281 

.005 .947 

This study demonstrates the feasibility of adopting ET and CBTs to replace conventional NS and PPTs. Under the 

same MC testing item construct, the researchers investigate the performance difference among examinees between 

partial scoring by the elimination testing and the conventional dichotomous scoring method. This study first builds a 

computer-based assessment system, then adopts experimental design to record an examinee’s performance with 

different testing tools and scoring approaches. One-way ANOVA does not show sufficient proof to reject the 

hypothesis that the performance of students taking CBTs is the same as the performance of students taking PPTs. 

This finding is in agreement with Alexander et al. (2001) and Bodman and Robinson (2004). We conclude that the 

discrepancy does not exist in the performance of examinees who take PPTs nor in the performance of those who take 

such CBTs when ET is used. 

Next, the correct number of NS was compared to the number of full knowledge of ET in each test. Data analysis 

demonstrates that the number of correct NS items is greater than the full knowledge number of ET for each test, 

which shows that ET can distinguish between full knowledge and partial knowledge by partial scoring. The number 

of unexpected responses calculated by the Rasch model based on item response theory for dichotomous scoring and 

the partial credit model based on graded item response for elimination testing to estimate the examinee ability and 

item difficulty parameters reveals that the number of unexpected responses in NS is greater than ET for each test, 

which demonstrates that ET reduces the unexpected responses of examinees. ET scoring is helpful whenever 

examinees’ partial knowledge and unexpected responses are concerned. Moreover, the instructors can more 

accurately assess examinees’ partial knowledge by adopting ET and CBTs, which is not only helpful in teaching, but 

also increases examinees’ eagerness and willingness to learn. 

Experimental results also indicate that different question content does not influence the performance of examinees on 

PPTs or CBTs. This result did not confirm Ager’s study (1993). The first reason might be the course content, an 

introductory course focused on principles and concepts of operations management, instead of on lots of complex 

calculation. The second explanation is that the subjects in this study are all students from the information 

management department. These students are quite used to interacting with computer interfaces; consequently, they 

present the difference of performance on PPTs and CBTs is insignificant. We remain aware that the validity of any 

experimental study is limited to the scope of the experiment. Since the study involved only two classes and one 


course, more comparison tests could be performed on other content subjects to induce those suited for adopting ET 

and CBTs to replace conventional NS and PPTs. 

Since the trend in e-learning technologies and system development is toward the creation of standards-based 

distributed computing applications. To maintain a high-quality testbank and promote the sharing and reusing of test 

items, further research should consider incorporating the IMS question and test interoperability (QTI) specification, 

which describes a basic structure for the representation of assessment data groups, questions, and results, and allows 

the CBT system to go further by using the shareable content object reference model (SCORM) and IMS metadata 

specification. 

References 

Abu-Sayf, F. K. (1979). Recent developments in the scoring of multiple-choice items. Educational Review, 31, 269– 

270. 

Ager, T. (1993). Online placement testing in mathematics and chemistry. Journal and Computer-Based Instruction, 

20 (2), 52–57. 

Akeroyd, F. M. (1982). Progress in multiple-choice scoring methods. Journal of Further and Higher Education, 6, 

87–90. 

Alexander, M. W., Bartlett, J. E., Truell, A. D., & Ouwenga, K. (2001). Testing in a computer technology course: An 

investigation of equivalency in performance between online and paper and pencil methods. Journal of Career and 

Technical Education, 18 (1), 69–80. 

Baker, F. B. (1992). Item response theory: Parameter estimation techniques, New York, NY: Marcel Dekker. 

Ben-Simon, A., Budescu, D. V., & Nevo, B. (1997). A comparative study of measures of partial knowledge in 

multiple-choice tests. Applied Psychological Measurement, 21 (1), 65–88. 

Bodmann, S. M. & Robinson, D. H. (2004). Speed and performance differences among computer-based and paperpencil 

tests. Journal of Educational Computing Research, 31 (1), 51–60. 

Bond, T. G. & Fox, C. M. (2001), Applying the Rasch model: Fundamental measurement in the human sciences, 

Mahwah, New Jersey: Lawrence Erlbaum Associates. 

Bradbard, D. A. & Green, S. B. (1986). Use of the Coombs elimination procedure in classroom tests. Journal of 

Experimental Education, 54, 68–72. 

Bradbard, D. A., Parker, D. F., & Stone, G. L. (2004). An alternate multiple-choice scoring procedure in a 

macroeconomics course. Decision Sciences Journal of Innovative Education, 2 (1), 11–26. 

Bugbee, A. C. (1996). The equivalence of PPTs and computer-based testing. Journal of Research on Computing in 

Education, 28 (3), 282–299. 

Burton, R. F. (2002). Misinformation, partial knowledge and guessing in true/false tests. Medical Education, 36, 

805–811. 

Bush, M. (2001). A multiple choice test that rewards partial knowledge. Journal of Further and Higher Education, 

25 (2), 157–163. 

Chan, N., & Kennedy, P. E. (2002). Are multiple-choice exams easier for economics students? A comparison of 

multiple-choice and “equivalent” constructed-response exam questions. Southern Economic Journal, 68 (4), 957– 

971. 


Coombs, C. H., Milholland, J. E., & Womer, F. B. (1956). The assessment of partial knowledge. Educational and 

Psychological Measurement, 16, 13–37. 

Dressel, P. L., & Schmid, J. (1953). Some modifications of the multiple-choice item. Educational and Psychological 

Measurement, 13, 574–595. 

Gretes, J. A., & Green, M. (2000). Improving undergraduate learning with computer-assisted assessment. Journal of 

Research on Computing in Education, 33 (1), 46–4. 

Hagler, A. S., Norman, G. J., Radick, L. R., Calfas, K. J., & Sallis, J. F. (2005) Comparability and reliability of 

paper- and computer-based measures of psychosocial constructs for adolescent fruit and vegetable and dietary fat 

intake. Journal of the American Dietetic Association, 105 (11), 1758–1764. 

Haladyna, T. M. & Downing, S. M. (1989). A taxonomy of multiple-choice item-writing rules. Applied Measurement 

in Education, 2, 37–50. 

Hambleton, R. K., & Swaminathan, H. (1985). Item response theory: Principles and applications, Boston, MA: 

Kluwer-Nijhoff. 

Inouye, D. K., & Bunderson, C. V. (1986). Four generations of computerized test administration. Machine-Mediated 

Learning, 1, 355–371. 

Jaradat, D. & Sawaged, S. (1986). The subset selection technique foe multiple-choice tests: An empirical inquiry. 

Journal of Educational Measurement, 23 (4), 369–376. 

Jaradat, D. & Tollefson, N. (1988). The impact of alternative scoring procedures for multiple-choice items on test 

reliability, validity, and grading. Educational and Psychological Measurement, 48, 627–635. 

Kehoe, J. (1995). Basic item analysis for multiple-choice tests. Practical assessment, Research & Evaluation, 4 (10), 

retrieved October 15, 2007, from http://PAREonline.net/getvn.asp?v=4&n=10. 

Linacre, J. M., & Wright, B. D. (1994). Chi-square fit statistics. Rasch Measurement Transactions, 8 (2), 360, 

retrieved October 15, 2007, from http://rasch.org/rmt/rmt82.htm. 

Linn, R. L., & Gronlund, N. E. (2000). Measurement and assessment in teaching (8 th Ed.), Upper Saddle River, NJ: 

Prentice-Hall. 

Mazzeo, J., & Harvey, A. L. (1988). The equivalence of scores from automated and conventional education and 

psychological test, New York, NY: College Board Publications. 

Wongwiwatthananukit, S., Popovich, N. G., & Bennett, D. E. (2000). Assessing pharmacy student knowledge on 

multiple-choice examinations using partial-credit scoring of combined-response multiple-choice items. American 

Journal of Pharmaceutical Education, 64 (1), 1–10. 

Wright, B. D., & Masters, G. N. (1982). Rating scale analysis, Chicago, IL: MESA Press. 

Wright, B. D., & Stone, M. H. (1979). Best test design, Chicago, IL: MESA Press. 

Zhu, W. (1996). Should total scores from a rating scale be used directly? Research Quarterly for Exercise and Sport, 

67 (3), 363–372. 

Zhu, W., & Cole, E. L. (1996). Many-faceted Rasch calibration of a gross motor instrument. Research Quarterly for 

Exercise and Sport, 67 (1), 24–34. 


Fleischmann, K. R. (2007). Standardization from Below: Science and Technology Standards and Educational Software. 

Educational Technology & Society, 10 (4), 110-117. 

Standardization from Below: Science and Technology Standards and 

Educational Software 

Kenneth R. Fleischmann 

College of Information Studies, University of Maryland, College Park, MD, USA // kfleisch@umd.edu 

ABSTRACT 

Education in the United States is becoming increasingly standardized, with the standards being initiated at the 

national level and then trickling down to the state level and finally the local level. Yet, this top-down approach 

to educational standards carries with it significant limitations, such as loss of local autonomy and restrictions on 

the creativity of educational software designers. This paper reports findings from a study of the design and use 

of frog dissection simulations used in middle school and high school biology classes. The paper builds on the 

existing literatures on science and technology standards in education, using interviews, participant observation, 

and content analysis guided by grounded theory. The results highlight the ways that top-down educational 

standards constrain science teachers and software designers. The discussion presents an alternative to the topdown 

regime of educational standards, namely, a bottom-up approach of standardization from below. Finally, 

the conclusion argues that local control of educational experiences in the form of standardization from below 

can improve upon the traditional regime of top-down standards. 

Keywords 

Information technology, Computers, Science standards, Technology standards, Frog dissection simulations 

Introduction: Top-Down Educational Standards 

Standardization in education, including the growth of both science and technology standards, is a major ongoing 

trend in education in the United States. Although educational standards have historically been created and enforced 

at the state level, the recent trend has been toward the creation and enforcement of national educational standards. 

The explicit goal of this standardization is to enable all students within the United States to use the same computers 

with the same educational software. Yet, the danger is that these standards represent a one-size-fits-all approach to 

education that may overlook the specific needs of local communities. Going too far in the direction of standardizing 

education may endanger more socially, culturally, and geographically appropriate education. This paper asks two key 

questions. First, how do current top-down science and technology standards influence the design, marketing, and use 

of educational software? Second, how might bottom-up science and technology standards, in contrast, differently 

influence on the design, marketing, and use of educational software? 

Prior to the 20th Century, town meetings, gossip, and small newspapers gave citizens a local context and a shared 

experience. In the 20th Century, new technologies such as the telephone, the radio, and the television expanded the 

shared experience of humanity. Now in the 21st Century, the Internet has the potential to shape this shared 

experience at a global level. At each of these stages, the standardization of shared experience has depended largely 

on available technologies. One important form of shared experience is the educational experience, such as the 

secondary school experience. Educational policies and technologies are now being used to standardize this 

experience from one locality to another, so that the transmission of knowledges from one generation to another can 

become increasingly uniform. The goal of this paper is to determine the advantages and disadvantages of this topdown 

approach to educational standards as well as to consider the possibility of an alternative approach, a bottom-up 

approach to educational standards. 

Background: Top-Down Science Standards 

Traditionally, educational content standards focused on the “Three R’s” of reading, (w)riting, and (a)rithmatic. High 

school exit examinations focused on these three areas. These examinations not only are required for graduation, but 

also may be used to determine funding for different schools, and may even be used to fire teachers or principals or 

even close high schools. As a consequence, English and mathematics were considered the most important areas of 

learning, and privileged above other domains such as the sciences, social sciences, and arts. Further, these standards 






110

were typically set at the state level, with relatively little involvement by the federal government, including the United 

States Department of Education. Yet, recently these situations appear to be changing. 

One important example of the growing importance of science standards is the National Science Education Standards, 

published in 1996 by the National Research Council. This document was publicized as the first nationwide list of 

science standards ever produced, and the primary goal was to increase the level of science understanding among all 

students. These national standards then trickled down to the state level, adding pressure on states to develop their 

own science standards in line with the National Science Education Standards. Indeed, as a direct result, “by the 

2007–08 school year, all states are expected to measure students' science knowledge at least once, at each level, in 

elementary, middle, and high school, under the federal ‘No Child Left Behind’ Act of 2001” (Hoff, 2002, p. 10). The 

“No Child Left Behind” Act further sped up the trickling down process for state science standards by two years, 

requiring that “by 2005-06, states must develop science standards” (National Education Association, 2003, p. 31). 

Thus, the National Science Education Standards and the No Child Left Behind Act were both responsible for 

promoting implementation of top-down science standards. 

How do state science standards work in practice? Since schools and school districts rely on states for budgets for 

textbooks, software, and other educational materials, the states generally have the power to enforce their standards, 

especially when they are connected to statewide exit exams. For example, one California school district devoted two 

years to developing science standards for grades K-12. Yet, just as the school district finished this process, the state 

of California published its own science standards, which was spurred by the National Science Education Standards. 

Since there were significant differences between the organization of the state and district science standards for grades 

6-8, the state demanded that the district immediately revise their standard to match the state standards, threatening to 

withdraw textbook purchasing support if the district did not immediately comply (Evans, 2002). Thus, the standards 

in this process began at the national level, then trickled down to the state level where they were enforced through the 

power of the purse strings, and finally trickled all the way down to the school district level where they overruled the 

local standards that had recently been developed. This story provides a clear example of top-down creation and 

enforcement of science standards. 

Background: Top-Down Technology Standards 

Technology standards are also an important consideration in educational software design, marketing, and use. 

Technology in general is particularly dependent on the notion of standards. Without standards, it may be difficult or 

impossible for different technologies to interface. In the case of education, some amount of technological 

standardization is necessary to guarantee that the same educational software can be used in different classrooms and 

computer laboratories (Owen, 1999). Tremendous amounts of money have been invested in improving the level of 

technology in schools to meet technical standards. Further, the implementation of technology standards can help all 

students, and thus may help teachers to bridge the Digital Divide between technological haves and have-nots (Swain 

and Pearson, 2002). 

Swain and Pearson’s (2002) analysis is particularly fascinating and compelling when they explore differences in 

students’ experiences with technology. They find that student ethnicity and socioeconomic status correlate to 

students’ experiences with technology. For example, they cite studies that demonstrate that “minority, poor, and 

urban students were more likely to use computers for lower-order thinking skills than their White, non-poor, and 

suburban counterparts” (p. 329). They use this finding to explain why increased access to technology does not 

always guarantee improved learning for students; learning depends not only on access to technology but also on the 

particular experiences with technology and the types of technology that are used. Similarly, Awalt and Jolly (1999) 

argue that technology standards would be useful for students, teachers, and administrators. 

Certainly, technology standards play an important role in educational settings. Yet, it is important to consider that the 

use of educational technology in schools encompasses both technological infrastructure and educational software, 

which is influenced by content as well as by the computers that run the software and the networks that connect the 

computers. While technology standards in education focus primarily on infrastructure, there may be value to 

considering standardization for educational software as a technology that goes beyond content standards such as 

science standards that currently have a top-down impact on the development and use of educational materials such as 

textbooks and educational software such as educational computer simulations. This paper studies the issue of 

111

educational standards in practice through studying how standards shape the design, marketing, and use of educational 

computer simulations. 

Research Methods 

This study was part of a larger dissertation project (Fleischmann, 2004) and other findings of this study have been 

published elsewhere (Fleischmann, 2003, 2005, 2006a, 2006b). Data from a variety of sources, including semistructured 

interviews, participant observation, and content analysis of software and promotional materials are used 

examine the impact of science and technology standards on educational software design, marketing, and use. The 

larger case study that informs this paper included 51 interviews, including 14 designers of six frog dissection 

simulations (some of whom currently are or previously were biology teachers), 29 users of frog dissection 

simulations (including three biology teachers, a principal, and 25 biology students), and 8 animal advocates who play 

a role in marketing dissection simulations. Participant observation consisted of fifteen days of fieldwork over a sixmonth 

period, including six days spent at three dissection simulation design laboratories, four days spent at a science 

education conference, and five days spent in a high school biology classroom. Print materials included promotional 

materials produced by dissection simulation manufacturers and animal advocacy organizations and state educational 

standards and other materials published by educational policymaking bodies. It is important to note that the scope of 

this study is limited almost entirely to the United States (although the study did discover some anecdotal evidence of 

United States educational standards influencing educational standards and software design abroad (specifically, in 

Canada). 

Data analysis for this study was based on the grounded theory approach to qualitative data analysis (Strauss & 

Corbin, 1998). Interviews were transcribed as soon as they were conducted, and each interview was coded according 

to a constantly changing set of categories that was initially based on the literature to date and then was continually 

modified as a result of new categories that emerged from interviews and old categories that lost their saliency as the 

data pointed to other categories. Specifically, the issue of standardization emerged as a result of the interviews, 

participant observation, and content analysis of software and promotional materials. Memos were written based on 

the coded interviews, and these memos then led to the development of theories that could best explain various 

aspects of the data. Specifically, the contrast between top-down and bottom-up standardization regimes was a result 

of this process. 

Results: Teachers’ Experiences with Standards and Educational Software Use 

The data collected for this study provide evidence that science and technology standards have a strong influence on 

the use of educational simulation software in biology classes, and this influence may be growing. As one teacher 

explains, “Everything that you’re doing will be aligned to the standards, or will be addressing the standards.” As one 

simulation designer and former teacher explains, “a teacher is looking for something that they can do their job with, 

and their job is to present those goals and objectives by the state or the nation.” Similarly, another simulation 

designer and former high school teacher argues, “as long as a teacher’s evaluation and salary is connected to the 

assessment tests, then what’s going to get taught is exactly what’s in the curriculum.” Thus, standards come with a 

strong motivation to teachers, not only to ‘do their job’ but also to continue to get paid for doing it. Another science 

teacher in a state that has recently begun to implement science testing as a requisite for graduation is concerned that 

this emphasis on testing may create more emphasis on textbook learning specifically geared toward improving test 

scores without activities such as hands-on laboratories or educational simulation use that might provide more 

innovative and engaging opportunities for learning. 

Perhaps the most interesting and vivid experience illustrating this point from the participant observation occurred 

during participant observation in the biology classroom. One day, the teacher began class by showing the students 

the state academic content standards for science (via the Web) and then explained, point by point, how the software 

that they were using, The Digital Frog 2, met specific standards in the area of physiology. Apparently, showing 

students the state standards is a typical activity for this teacher, since it helps to illustrate why it is important to learn 

the material. This example illustrates that not only are teachers aware of the state standards that they must meet, but 

they may also explicitly inform their students of these standards and use them to defend their use of educational 

software, in this case as a replacement for a hands-on laboratory activity, frog dissection. This state is also in the 

112

process of requiring graduating students to pass a science exam, making the issue of meeting educational standards 

particularly compelling and current. 

The emphasis on science standards on schools in the United States has a significant impact in shaping the content of 

all curricular materials, ranging from textbooks to educational software. As a teacher asks, “if it’s not addressing the 

main topics of the curriculum, which are the standards, then what is the point?” While textbooks were the original 

model for this, the same forces are now shaping educational software adoption. A teacher explains: 

Nobody these days will buy a book that’s not aligned [to state standards], and I see the same thing 

could definitely happen with software, although it doesn’t seem to be at that level yet because 

software is not as widely implemented as textbooks, but that will probably change too. So I think the 

companies would do well – it would serve their clientele better to align the curriculum with their 

software – but states have different standards. 

Interestingly, as one informant pointed out, this leads to a major problem in terms of equity – large states 

traditionally run the textbook industry, and may now have the same impact on the educational software industry, 

because adoption by large states can determine whether or not a textbook is successful. Smaller states, then, tend to 

follow the lead of the large states, and are not able to have as much of a direct impact on textbook and educational 

software adoption. Thus, the reliance on state standards has the effect of privileging large states while 

disempowering smaller ones. 

Given the constraints they are under, teachers seem to naturally gravitate toward materials that make explicitly clear 

how they correspond to state standards. For example, a teacher comments, “I think it would be awesome if the 

materials were provided with the standards links embedded, and some materials are.” The teacher continues, “I 

would think that software companies would do enormously well to produce software…that addresses the standards 

and make sure that it’s targeted to the standards.” The teacher then provides the following example: 

For me as a teacher, if I was to purchase this particular software…the superintendent and the board, 

they will not approve it unless I can show that it’s aligned. I would choose software that already says 

to me, it’s aligned; these are the standards that are addressed and here’s where they are addressed in 

the package. Then I can present it and say, look, it’s aligned already, and here it is. If that’s work that I 

have to do, if I have to go through and find the links, then honestly I would rather find one that’s 

easier to present and easier to get approved. 

This hypothetical example clearly illustrates why it is important, under the current educational standards regime, for 

educational software to meet state standards and explicitly demonstrate how their software meets various state 

standards, at least, as noted above, for the large states that play the most powerful role in influencing nationwide 

textbook and educational software adoption. 

An interesting feature of the high school, which was located in a very rural area, was the emphasis on technology. 

The high school had fairly sophisticated computing resources, and placed significant emphasis on these resources. 

The biology teacher and principal frequently referred to their high school as a “Digital High School,” since it was 

supported by a state educational initiative of the same name that provided technology resources to high schools 

across the state. As part of this program, the school must continue to make good use of these technology resources, 

and both the teacher and the principal explicitly explained that using dissection simulation software was one way to 

meet this requirement. The teacher and principal also emphasized to me that the computer use at the high school 

would help prepare students for the job market, since despite its rural surroundings, the high school was still 

relatively proximate to a major high-tech center. Educational simulation software use helps schools to spotlight and 

make good use of their technology resources provided as a result of technology standards. 

Use of simulations is also often connected to technology standards that control the types of computers available in 

computers as well as the student-to-computer ratio (Fleischmann, 2005). Specifically, the student-to-computer ratio 

is important for science software design. One of the key findings of this study published elsewhere (Fleischmann, 

2005) was that although an assumption is typically made based on the conventional understanding of humancomputer 

interaction that students should each get their own computer and work individually to complete computerbased 

tasks, this social environment differs from the traditional K-12 dissection experience and other science 

113

laboratory activities, which are built upon interaction among students working in small groups. As a result of 

transforming the learning process in science from a richly interactive and social environment to a solitary activity, 

students have less opportunity for peer learning and social support. As a result, the overall educational experience 

may suffer, an outcome presumably not intended or expected by educational software designers and technology 

standard-setters. 

Results: Designers’ Experiences with Standards and Educational Software Design and 

Marketing 

Science and technology standards also have a strong and growing influence on the design and marketing of 

educational simulation software. Indeed, this influence is so strong that a product may succeed or fail due largely to 

standards. A software designer explains, “Every software manufacturer that does anything with client software in 

particular has to look at goals and objectives [contained in science standards]. Maybe on a state level, maybe on a 

national level.” One particularly compelling example is of a software product that failed because of national and state 

standards. In this case, the product focused on a topic that was not covered within most state standards, and as a 

result, it was doomed to failure. As one of the designers explains, “we found out after that product that if our 

simulations aren’t part of the standards and part of the standard course of study, then there’s not going to be much 

interest in it.” Thus, when designing simulations, designers must be mindful to match the content of their simulation 

to existing educational standards. 

Another educational software design company was also influenced by science standards in its development of 

educational simulations. After finishing their first simulation, a frog dissection simulation, the company began 

developing a series of digital field trips, which explored various ecosystems. Yet, the company made a major 

strategic readjustment, moving from digital field trips (including canceling a half-completed product) and instead 

chose to focus on curriculum-specific units to meet particular standards. In this case, the standards seem to have 

robbed the software designers to some extent of their creativity, and certainly of their intellectual freedom to develop 

content, since they must develop content specifically to meet standards. 

Educational standards also have a significant impact on the marketing of educational simulation software. For 

example, the website of one of the studied educational software manufacturers includes a page where teachers and 

administrators can find out how the company’s software meets specific science standards. The page includes the 

science standards of eight U.S. states (including four of the five largest states in the US) and two Canadian 

provinces. For nine of these states and provinces, the company has developed their own page, but in the case of 

California, the site links to an external page developed by the California Learning Resource Network which 

evaluates the company’s simulations and how well they match up with the California’s science standards. The 

California Learning Resource Network was established by the California Department of Education to evaluate 

educational software and its alignment to state academic content standards. It has reviewers that provide the 

information about particular software. Software must be submitted for review, and this particular software package is 

apparently the only frog dissection simulation that has been submitted for review by its designers. Thus, educational 

software manufacturers can conduct their own analyses of its software’s compliance with state and province 

standards and also submit their software for review to websites such as the California Learning Resource Network. 

Teacher conferences are a good opportunity for simulation designers to interact directly with teachers. Frog 

dissection simulation designers routinely attend national, state, and local science teacher conferences in an effort to 

market their simulation products and boost their sales. Animal advocates representing various organizations also 

attend these conferences. The primary message of the animal advocates is that dissection simulation software and 

other alternatives to dissection can meet science standards as well as or better than the practice of wetlab dissection. 

So, at teacher conferences, not only do dissection simulation designers argue that their products cover educational 

standards, but so do animal advocates. Further, animal advocates, both in their print materials and in the interviews, 

emphasize that state science standards do not contain any explicit requirement for students to participate in animal 

dissection, and that in contrast, many states have passed dissection choice laws requiring teachers to provide 

alternatives such as dissection simulations to students who object to the activity of animal dissection on moral or 

ethical grounds (Fleischmann, 2003). Interviews with students revealed a variety of perspectives on the issue of 

alternatives to dissection, ranging from emphasis on student choice to a preference for required dissection. 

114

Design and marketing of simulations in classrooms are also tied to technology standards. For many designers, it was 

their interest in technology and the growth of technology use that interested them in simulation design, rather than 

the particular content of the simulations that they participated in designing. Further, without the increased availability 

of computers in schools brought about in part by technology standards, they would not have an adequate market for 

their products. In the marketing of educational simulations, an emphasis on the importance of learning about and 

with technology is often present in the literature of dissection simulation manufacturers as well as their allies in the 

animal advocacy movement (Fleischmann, 2003). 

The most compelling quote relevant to the issue of standards in educational simulation design was provided by a 

simulation designer and former teacher, who argues “The selection of the curriculum ought to reflect the local 

community…with a state-wide curriculum, there are needs that are not addressed or overly addressed in different 

areas of the state.” This teacher argues that standards may be a step too far in the wrong direction – instead of 

standardizing everything, teachers should have the ability to tailor their curriculum to meet the needs of their local 

school district. Certainly, this issue has direct relevance on the design of educational software, since educational 

standards reduce not only teachers’ ability to meet the specific needs of their students but also designers’ ability to 

innovate and produce products that are relevant to particular target audiences. 

Discussion: Top-Down Educational Standards Versus Local Knowledges 

What are the motivations for implementing standards? According to Feng (2002), standards historically have been 

used to achieve several different goals, including consistency and efficacy. These motivations may occur in the case 

of educational standards such as science and technology standards. Feng argues that consistency is used to make 

arguments that standards can serve the cause of social justice, by serving as an equalizer. Swain and Pearson (2003) 

make a similar argument about technology standards. However, Feng cautions that efficacy of standardization leads 

to hegemony, as a form of uniformity from above. The potentially hegemonic nature is illustrated in the example 

provided by Evans (2002) above, where the state board of education uses standards to dictate content to school 

districts and schools. This power relationship is also present in the data provided above, such that teachers must 

follow state standards and software developers must produce products that not only follow state standards but also 

explicitly demonstrate which standards they address. 

Do standards serve social justice? Swain and Pearson (2002) make a convincing argument that they do, since they 

can help to ensure that all schools, teachers, and students have equal access to equipment, training, and software that 

builds higher-order thinking skills. Yet, there is a danger that, swooping in from above, standards may ignore the 

social, cultural, and geographic context in which the students are learning. Monahan (2005a, 2005b) cautions, in 

some cases, that “the question of where standards are set and by whom determines where power is shifting to, on the 

one hand, and where autonomy is lost, on the other” (2005a: 601). When seen in this way, standards from above can 

be seen as endangering local control of educational experiences. The final quote provided in the results section above 

most clearly makes this point. 

According to Geertz (1983) and Hess (1995), all knowledges are socially, culturally, and geographically situated. 

Hess relies on the case of medicine, and discusses the various non-Western medical traditions that have evolved, like 

biomedicine, over long periods of time, and which, at least in many cases, are becoming increasingly popular in the 

contemporary United States. Eglash (1999) demonstrates that local knowledges can also apply to areas such as 

mathematics. He finds that many African cultures developed a deep understanding of fractal geometry long before it 

was “discovered” in the West. He then puts this research into practice by using it to encourage achievement in 

mathematics by African-American students. In his current research, he makes a similar intervention into the teaching 

of Native American students. Thus, local knowledges can be useful for stimulating interest in an educational subject, 

especially among specific social, cultural, and geographical groups. While standards clearly have many positive 

benefits as described above, they may also reduce the autonomy and specificity of local educational curricula when 

they are created and implemented through a largely top-down process. 

Conclusion: A Bottom-Up Approach to Science and Technology Standards 

Science and technology standards play a significant role in the design, marketing, and use of educational software. 

Certainly, there are benefits to science and technology standards, as discussed above. Yet, the standardization 

115

process is currently a largely top-down process, with design occurring primarily at the national and state level while 

implementation takes place at the local level with pressure applied from above, as explained by Hoff (2002) and 

Evans (2002). As demonstrated here, top-down state science standards have the effect of stifling innovation in 

software design, as in the case of two of the simulation design companies described above. Similarly, technology 

standards may not take into consideration the specific needs of local communities. After examining the impacts of 

the current top-down regime of science and technology standards on educational software design, marketing, and use 

in practice, it seems useful to consider an alternative approach to standard-setting, a bottom-up approach. 

Is a top-down process the only or even the best way of designing and implementing science and technology 

standards? Standards-setting processes are often promoted as participatory, with input being sought from teachers 

and lower-level administrators. Yet, this is still a top-down approach, since standards are first set at the national or 

state level and then trickle down to districts and schools who are compelled, often through incentives such as 

standardized testing and the purchasing of texts, software, and other equipment, to adopt the dominant standards. 

Further, at the state level, it is the large states that have the most impact on textbooks and software, creating an 

inequality of fairness among the states. A bottom-up approach to educational standards would replace the centralized 

power of standard-setting bodies at the national level and within large states with a more diffuse power that is spread 

more evenly among schools and school districts, giving them more autonomy to control their own classroom content 

and giving software developers more room to innovate to meet the diverse needs of local schools. 

Perhaps it would be useful to examine not only the effectiveness of this structure but also the potential of inverting it 

to allow for more local control of educational content and equipment. In such a scenario, schools and districts might 

begin the standard-setting process, which would then be built up to the state and then the national level as a process 

of consensus-building. Local standards-setters could get the input of larger educational bodies in an advisory role, 

rather than the reverse. A process of standardization from below might allow teachers, students, and administrators to 

reap the benefits of standardization discussed above while still retaining local autonomy and control over content, 

and leaving the door open for more innovation in educational software design, marketing, and use. By empowering 

teachers to serve not only as software designers (Fleischmann, 2006a) but also as standards-setters, it would allow 

them to control the content that they teach rather than merely implementing the wills of faceless standards boards, 

ensuring that they would be able to meet the real needs of the students in their classrooms. 

Acknowledgements 

Thanks go to David J. Hess, Bo Xie, and three anonymous reviewers for reading and commenting on earlier drafts of 

this paper, as well as to all of the interviewees (named and anonymous) who participated in this study. This study 

was funded by a Dissertation Research Improvement Grant from the Science and Society Program of the National 

Science Foundation (SES-0217996). 

References 

Awalt, C., & Jolly, D. (1999). An inch deep and a mile wide: Electronic tools for savvy administrators. Educational 

Technology & Society, 2 (3), 97-105. 

Eglash, R. (1999). African fractals: Modern computing and indigenous design, New Brunswick, NJ: Rutgers 


Evans, S. M. (2002). Aligning to state standards. Science Teacher, 69 (3), 54-57. 

Feng, P. (2002). Designing a “global” privacy standard: Politics and expertise in technical standards-setting, 

Unpublished doctoral dissertation, Rensselaer Polytechnic Institute, Troy, NY. 

Fleischmann, K. R. (2003). Frog and cyberfrog are friends: Dissection simulation and animal advocacy. Society and 

Animals, 11 (2), 123-143. 

116

Fleischmann, K. R. (2004). Exploring the design-use interface: The agency of boundary objects in educational 

technology, Unpublished doctoral dissertation, Rensselaer Polytechnic Institute, Troy, NY. 

Fleischmann, K. R. (2005). Virtual dissection and physical collaboration. First Monday, 10 (5), retrieved October 15, 

2007 from http://www.firstmonday.org/issues/issue10_5/fleischmann/index.html. 

Fleischmann, K. R. (2006a). Do-it-yourself information technology: Role hybridization and the design-use interface. 

Journal of the American Society for Information Science and Technology, 57 (1), 87-95. 

Fleischmann, K. R. (2006b). Boundary objects with agency: A method for studying the design-use interface. The 

Information Society, 22 (2), 77-87. 

Geertz, C. (1983). Local knowledge: Further essays in interpretive anthropology, New York: Basic Books. 

Hess, D. J. (1995). Science and technology in a multicultural world, New York: Columbia University Press. 

Hoff, D. J. (2002). Science standards have yet to seep into class, panel says. Education Week, 21 (37), 10. 

Monahan, T. (2005a). The school system as a post-Fordist organization: Fragmented centralization and the 

emergence of IT specialists. Critical Sociology, 31 (4), 583-615. 

Monahan, T. (2005b). Globalization, Technological Change, and Public Education, New York: Routledge. 

National Education Association (2003). Timeline for future NCLB mandates. NEA Today, 21 (8), 34. 

National Research Council (1996). National Science Education Standards, retrieved October 15, 2007 from 

http://books.nap.edu/readingroom/books/nses/. 

Owen, M. (1999). Appropriate and appropriated technology: technological literacy and educational software 

standards. Educational Technology & Society, 2 (4), 62-69. 

Strauss, A. & Corbin, J. (1998). Basics of qualitative research: Techniques and procedures for developing grounded 

theory (2 nd Ed.), Thousand Oaks, CA: SAGE Publications. 

Swain, C., & Pearson, T. (2002). Educators and technology standards: Influencing the Digital Divide. Journal of 

Research on Technology in Education, 34 (3), 326-335. 

117

Hsu, Y.-S., Wu, H.-K., & Hwang, F.-K. (2007). Factors Influencing Junior High School Teachers’ Computer-Based Instructional 

Practices Regarding Their Instructional Evolution Stages. Educational Technology & Society, 10 (4), 118-130. 

Factors Influencing Junior High School Teachers’ Computer-Based 

Instructional Practices Regarding Their Instructional Evolution Stages 

Ying-Shao Hsu 

Department of Earth Sciences & Science Education Center, National Taiwan Normal University, Taiwan // 

yshsu@.ntnu.edu.tw 

Hsin-Kai Wu 

Graduate Institute of Science Education, National Taiwan Normal University, Taiwan // hkwu@.ntnu.edu.tw 

Fu-Kwun Hwang 

Department of Physics, National Taiwan Normal University, Taiwan // hwang@phy.ntnu.edu.tw 

ABSTRACT 

Sandholtz, Ringstaff, & Dwyer (1996) list five stages in the “evolution” of a teacher’s capacity for computerbased 

instruction—entry, adoption, adaptation, appropriation and invention—which hereafter will be called the 

teacher’s computer-based instructional evolution. In this study of approximately six hundred junior high school 

science and mathematics teachers in Taiwan who have integrated computing technology into their instruction, 

we correlated each teacher’s stage of computer-based instructional evolution with factors, such as attitude 

toward computer-based instruction, belief in the effectiveness of such instruction, degree of technological 

practice in the classroom, the teacher’s number of years of teaching experience (or “seniority”), and the 

teacher’s school’s ability to acquire technical and personnel resources (i.e. computer support and maintenance 

resources). We found, among other things, that the stage of computer-based instructional evolution and teaching 

seniority, two largely independent factors, both had a significant impact on the technical and personnel 

resources available in their schools. Also, we learned that “belief” in the effectiveness of computer-based 

instruction is the single biggest predictor of a teacher’s successful practice of it in the classroom. Future research 

therefore needs to focus on how we can shape teachers’ beliefs regarding computer-based learning in order to 

promote their instructional evolution. 

Keywords 

Technology adoption, Teachers’ beliefs, Educational technology, In-service teachers 

Introduction 

The rapid development of modern information and communication technologies has opened new possibilities for 

establishing and delivering distance learning. Given the popularity of the Internet, computer applications have 

recently become one of the most promising kinds of educational tool. Computers can now help educators in 

designing and promoting the teaching and learning process (Ministry of Education in Taiwan, 1999; Sinko & 

Lehtinen, 1999; Smeets, Mooij, Bamps, Bartolomé, Lowyck, Redmond, & Steffens, 1999). From studies (Angeli & 

Valanides, 2005; Hsu, Cheng , & Chiou, 2003), computers or/and Internet technology have positive impacts on 

students learning only when teachers know how to use computers or/and Internet technology to promote students’ 

knowledge construction and thinking. How can teachers use computers or/and Internet technology to promote 

students’ meaningful learning? 

Firstly, the teacher’s role should no longer be that of a traditional lecturer; rather, the teacher must now be a coach or 

co-learner (Beaudion, 1990; Brophy & Good, 1986). Secondly, activities in the classroom should become learnercentered 

and flexible in order to help students organize information and undergo self-initiated, exploratory learning 

processes (McKenzie, Mims, Davidson, Hurt & Clay, 1996; Winn, 1993). With computers and Internet technology, a 

teacher can utilize online teaching resources to arrange flexible learning activities; these can assist students in 

analyzing and organizing large amounts of information. Thirdly, the teacher’s attitude toward computers will be 

important to the way computer-based technology is used in instruction (Beaudion,1990; Ercan & Ozdemir, 2006; 

Gardner ,Discena & Dukes, 1993). Lloyd and Gressard (1984) have pointed out that a teacher’s positive feelings 

about computers will also help to generate or reinforce such feelings in the students. Comber et al. (1997) found that 

younger teachers might have more experience in computer use and thus a more positive attitude toward computers 

(Jennings & Onwuegbuzie, 2001). Braak (2001) noted that personal acceptance of technological innovation would 






118

influence attitudes toward computers, and furthermore that computer experience tends to directly affect attitudes 

toward computers (Kay, 1989; Gardner, Discena & Dukes, 1993; Woodrow, 1994; Yildirim, 2000). 

The teachers will need to adapt to the impact of computing technology while integrating it into their classrooms. 

During this adaptation process, a teacher may need to change his educational beliefs, values, and rationale in order to 

properly integrate computing and Internet technology into daily teaching. A researcher who wants to understand a 

teacher’s adaptation process should investigate the changes in the teacher’s teaching strategies, the design of his/her 

learning activities, and students’ assessments (Beaudoin, 1990; Brophy & Good, 1986; Mc Kenzie, Mims, Davidson, 

Hurt & Clay, 1996; Thatch & Murphy, 1995; Verduin & Clark, 1991). Therefore, integrating technology into 

instruction not only requires dealing with hardware and software issues but also dealing with complex issues like 

human cognition, policy and values (Sandholtz, Ringstaff, & Dwyer, 1996). For instances, teachers’ knowledge 

(OTA, 1995; Pelgrum, 2001), educational rationales (Czerniak & Lumpe, 1996; Niederhauser & Stoddart, 2001; 

Ruthven Hennessy & Brindley, 2004) and instructional strategies (Becker, 2000a, 2000b; Ravitz, Becker & Brindley, 

2000) about integrating computers into instruction affect students’ meaningful learning with computers. Educational 

policy, curriculum standards, school culture and peer supports influence teachers’ intention to use computers in 

classrooms (Chiero, 1997; Mooij & Smeets, 2001; Rogers, 2000; Russell, Bebell, O’Dwyer & O’Connor, 2003; 

Ruthven et al., 2004; Windschitl & Sahl, 2002; Teo & Wei, 2001; Zhao & Frank, 2003). Above all, the key to 

successful computer-based instruction, especially with teachers who are new to it, is to find the methods which can 

help teachers face and adjust their beliefs, attitude, and instructional strategies regarding computer-based instruction. 

From the above studies, a comprehensive list of factors influencing computer-based instruction has been addressed 

but there is a systematical investigation necessary to reveal the interactions among factors and the possibility of 

incorporating computers into a broader educational reform context. This study takes structural factors such as 

teachers’ instructional evolution and teaching seniority into consideration and explores how they interact with 

teachers’ beliefs regarding, attitude toward and practices of computer-based teaching. We conducted a national 

survey in Taiwan and collected a pool of six hundred science and mathematics teachers who had integrated 

computing technology into their instruction in junior high schools. The following questions guided this study: (1) 

What are the numbers of teachers in different stages of instructional evolution as defined by Sandholtz, Ringstaff and 

Dwyer (1996)? (2) How do teachers’ instructional evolution and teaching seniority interact with their beliefs 

regarding, attitudes toward and practices of computer-based instruction? (3) What regression models can be proposed 

to examine the relationships among teachers’ beliefs, attitudes, practices, and resources when using computing 

technology in the classroom? 

Research Methods 

This research project employed a survey method to investigate teachers’ beliefs regarding, attitudes toward and 

practices of computing technology in the classroom. 

Sample selection 

The population under investigation included all science and mathematics teachers at junior high schools in Taiwan. 

A stratified cluster random technique was employed to select the sample according to school size (large: more than 

37 classes, middle: 16-36 classes, small: less than 15 classes) and school region (schools in the northern, middle, 

southern, and eastern parts of Taiwan). Among 892 junior high schools in Taiwan, approximately 11% (99 schools) 

were selected. Of 2019 questionnaires mailed out, 1002 replies from 82 schools were received (an 89.1% school 

response rate and 49.6% teacher response rate). After ignoring the questionnaires which were not filled in 

completely, the valid sample size was determined to be 940. Of the valid sample, 613 respondents who reported that 

they had used computing technology in their teaching were finally examined. 

Instrumentation 

We developed the questionnaire to collect information on teachers’ use of computing technology in their classrooms. 

The questionnaire was divided into three sections: (1) demographic background, (2) current stage of instructional 

119

evolution with computing technology, and (3) perceptions and practices of computer-based instruction. Demographic 

background included information about age, gender, teaching seniority, school size and school region. Items in the 

latter two sections were rated on a 5-point Likert-type scale from 1 (strongly disagree) to 5 (strongly agree). From 

data in the second section, the respondents were classified into five evolutionary stages (entry, adoption, adaptation, 

appropriation and invention) to indicate teachers’ level of computer use, according to the definitions of Sandholtz, 

Ringstaff, & Dwyer (1996): (1) entry stage: teachers spend a lot of time in installing software and managing 

hardware and students spend most of time in learning computer skills instead of subject contents; (2) adoption stage: 

teachers utilize software (i.e. word processors, excel etc.) to assist their traditional teaching; (3) adaptation stage: 

teacher apply various software for instructional purposes and integrate technology successfully in 

classrooms; (4) appropriation stage: teachers develop multiple teaching strategies to promote students’ cognitive 

ability, share computer-based teaching experience with other teachers, and feel confident in integrating technology 

into teaching; (5) invention stage: teachers lead students to use software as a learning tool, develop innovative 

teaching strategies and assessments with computers, and affirm the value of computer-based instruction. 

Table 1 shows teachers’ instructional evolution with computers in terms of five categories: classroom management, 

software use, teaching strategies, learning efficiency, and confidence and beliefs. There is an item for each of the five 

“evolutionary” stages plotted against the five categories: thus 25 items in all. The respondents needed to pick one 

description for each category to represent their current level of experience with computing technologies. The round 

average of the values (from 1: entry to 5: invention) in these five categories represents the current stage of the 

teacher’s computer-based instructional evolution. 

Contractors 

Table 1. Characteristics of the Stages of Instructional Evolution (Sandholtz, Ringstaff, & Dwyer, 1996) 

Stages 

Classroom 

management 

Software use 

Teaching 

strategies 

Learning 

efficiency 

Confidence 

and beliefs 

Reacting to 

problems 

Entry Adoption Adaptation Appropriation Invention 

Dealing with 

problems in 

software 

installations and 

management 

No change 

Students spend 

time in learning 

computer skills 

No faith in 

computer- 

based 

instruction; 

having doubts 

most of the time 

Anticipating and 

developing strategies 

for solving problems 

Learning software 

Designing activities to 

teach students 

computer skills 

Promoting learning 

motivation but not 

improving conceptual 

understanding 

(sometimes having a 

negative impact on 

students’ grades) 

Attempting to use 

technology in 

classrooms 

Utilizing the 

technological 

advantage in 

managing the 

classroom 

Using software for 

instructional 

purposes 

Integrating 

technology to 

improve students’ 

knowledge 

comprehension 

Reducing students’ 

learning load and no 

significant 

improvement in 

conceptual 

understanding 

Often integrating 

technology 

successfully in 

classrooms 

Intertwining instruction approaches and 

management strategies 

Integrating software 

in learning 

processes and 

enhancing students’ 

mutual support in 

software use 

Using multiple 

methods to promote 

students’ cognitive 

ability 

Cultivating 

cognitive ability 

Having confidence 

in integrating 

technology into 

teaching; sharing 

experiences with 

other teachers 

Leading students 

to use software 

as a learning tool 

Developing 

strategies for 

innovative 

teaching such as 

project-based 

learning, 

modeling etc. 

Promoting 

problem- 

solving ability 

Affirming the 

value of 

computer-based 

instruction 

120

In the third section, factor analytical techniques were used to determine the underlying structure of teachers’ 

responses to items. Principal axis factor analysis with varimax rotations was employed for the factor analysis. The 

results for both the Kaiser-Meyer-Olkin Measure of Sampling Adequacy (0.85) and the Bartlett Test of Sphericity 

(χ2 =6803.8, N = 613, p < 0.0001) were significant, indicating that factor analysis was suitable for this sample. By 

using Cattell’s screen test and examining the factor loadings of the items, we removed 13 items from the 

questionnaire: five factors emerged. According to the pattern of correlation between and among items and factors, 

we assigned a descriptive name to each of the factors. The factors arrived at were: (1) belief in the use of computing 

technology in classrooms (e.g., “I believe that technology-based instruction can improve learning achievement”); (2) 

high degree of interactive use of technology (e.g., “I have had students learn collaboratively through the Internet”); 

(3) technical and personnel resources available in a given teacher’s school (e.g., “In my school, there are enough 

technicians to maintain computers”); (4) low degree of interactive use of technology (e.g., “I have used computers to 

play videos in classrooms”); and (5) attitude toward technology-based instruction (e.g., “Learning software will not 

make me nervous and uncomfortable”). The validity values of the factors (eigen values) were 4.82, 2.13, 1.73, 1.24, 

and 1.01. The factor loadings of the 24 items ranged from 0.53 to 0.70, and 58% of the variations were included 

within the five factors (see details in Table 2). As shown in Table 2, the composite reliability coefficients were 

ranged from 0.69 to 0.89 and the overall instrument reliability reached 0.87. Therefore, the reliability of the 

instrument was established. 

Data Analysis 

We used frequency analysis to show the distribution of teachers in different stages of instructional evolution and 

MANOVA techniques to examine correlations and interactions between two factors (teaching seniority and the stage 

of computer-based instructional evolution) on five measures (beliefs, high-level interactive practices, technical and 

personnel resources, low-level interactive practices, and attitudes). The stepwise method of multiple linear regression 

was applied to indicate the relationships between and among these variables. The calculations to determine the 

coefficients of regression models and MANOVA were performed through the application of the SPSS 12.0 package. 

No. Items and Factors 

Table 2. Questionnaire Items and Factor Loading 

Factor 

Loading 

Average 

variance 

extracted 

Composite 

reliability 

Subscale score 

Mean S.D. 

Belief 0.53 0.89 

e19 I think technology is helpful for my teaching. 0.73 3.60 0.73 

e26 I believe that technology-based instruction can improve 

learning achievement. 

0.71 3.51 0.78 

e9 I think technology-based instruction is one of the future trends 

in education. 

0.69 3.93 0.76 

e27 I believe that technology-based instruction can make my 

teaching more lively and energetic. 

0.68 3.89 0.67 

e21 I should create different teaching strategies for technologybased 

instruction. 

0.68 3.74 0.67 

e25 I believe that technology-based teaching can increase students’ 

motivation. 

0.68 3.81 0.66 

e20 Using technology can help me share my teaching experiences 

with others. 

0.68 3.59 0.68 

e10 I am willing to follow school policy on implementing 

technology-based instruction. 

0.66 3.80 0.71 

e22 I should develop different assessment strategies for technologybased 

instruction. 

0.63 3.72 0.68 

e1 I believe that conventional teaching methods are more efficient 

than technology-based instruction. 

0.54 3.35 0.85 

High-interaction practices (behaviors) 0.70 0.84 

e35 I have designed activities that allow students to learn through 

0.84 2.97 1.04 

the Internet. 

e36 I have had students learn collaboratively through the Internet. 0.84 2.86 1.02 

e37 I have used the Internet to support individual learning. 0.80 2.66 1.01 

e33 I have used computers and the Internet to collect and grade 

students’ assignments. 

0.72 3.24 1.09 

121

Resources 0.55 0.72 

e17 In my school, there are enough technicians to maintain 

computers. 

0.83 3.06 1.01 

e16 In my school, administrators can provide hardware and 

software for supporting technology-based instruction. 

0.77 3.40 0.90 

e13 In my school, teachers often discuss computer-related topics 

and exchange ideas about computer hardware and software. 

0.65 3.03 0.87 

e14 In my school, teachers often surf teaching websites. 0.56 3.55 0.77 

Low-interaction practices (behaviors) 0.56 0.72 

e32 I have used educational software to promote learning. 0.74 3.54 0.95 

e30 I have used computers to play videos in classrooms 0.70 3.55 0.96 

e34 I have used computer applications to create pictures, videos and 

animations and used them in classrooms. 

0.61 3.55 0.92 

e31 I have used document management software, such as Word and 

Powerpoint, to display my syllabus and my lectures. 

0.58 4.02 0.79 

Attitude toward computer technology 0.62 0.69 

e3 I will not feel anxious when I take any computer-related 

courses. 

0.87 3.61 0.97 

e2 Learning software will not make me feel nervous and 

uncomfortable. 

0.82 3.67 0.84 

e5 Currently, information on the Internet is useful to my teaching. 

Instrument reliability: 0.87 

0.51 3.71 0.80 

Results and Discussion 

The results are presented in three sections. The first section shows the descriptive statistics associated with two 

factors (stage of computer-based instructional evolution and teaching seniority) and five dependent measures 

(beliefs, high-interaction practices, low-interaction practices, resources and attitudes). In the second section, results 

of MANOVA are presented and interactive effects between/among the stage of instructional evolution and teaching 

seniority in relation to the five dependent measures are shown. The third section outlines the relationships among 

beliefs, high- and low-interaction practices, resources and attitudes. 

Descriptive analyses 

According to the participants’ responses in the second section of the questionnaire (current stage of instructional 

evolution with computing technology), one-third of the teachers (about 37%) were in the third or “adaptation” stage; 

the teachers in that stage can use appropriate software and information technology to improve their teaching and 

students’ learning in science and mathematics. Roughly one-fifth of the teachers were in either the “entry” stage 

(15%), “adoption” stage (23%) or “appropriation” stage (18%). Very few teachers (about 7%) were in the final 

“invention” stage; the teachers in this stage need to be able to use computer-based instruction in a creative way, 

guiding students to use software as a learning tool in their knowledge construction, modeling, and communication. 

After surveying junior high school science and mathematics teachers who had integrated computing technology into 

their instruction in Taiwan, most of them were in adaptation and appropriation stages; that meant they usually 

applied computer software for instructional purposes, developed multiple teaching strategies to promote 

students’ cognitive ability, and felt confident in integrating technology into teaching. 

Teachers’ degree of seniority could indicate the degree of their teaching experiences, pedagogical content knowledge 

(PCK), subject knowledge, and computer skills. Teachers with a seniority of less than 10 years tended to have more 

computer skills because they more likely were able to take computer-based instruction courses in their teachers’ 

training programs. In contrast, the teachers who had already taught for more that 11 years tended to lack training in 

computer skills and to feel anxious about computers, while also having a higher degree of PCK due to their greater 

teaching experience. As Figure 1 shows, most teachers with low seniority are in the third or “adaptation” stage: 43% 

of the teachers who have taught for less than 5 years are in this stage. This means that most of them often integrate 

technology into their instruction but cannot use multiple methods to promote students conceptual understanding and 

cognitive ability because they lack the teaching experience, even though their computer skills are probably excellent. 

About 76% of this low-seniority group are in the first three stages (entry, adoption, and adaptation). On the other 

122

hand, more teachers whose seniority ranged between 16 and 20 years were in the fourth or “appropriation” stage 

(25%). This means teachers with more teaching experience and greater PCK can use multiple methods to integrate 

technology into their classroom teaching; however, with greater seniority the percentage of teachers in the “entry” 

and “adoption” stages increased as well, suggesting again that younger teachers’ greater computer skills and more 

positive attitudes toward computers do also affect computer-based instructional evolution . 

Figure 1. Frequencies of teachers’ seniorities at each evolutionary stage 

Effects of instructional evolution and teaching seniority 

Table 2 outlines the mean scale scores and standard deviations for different -stage and different-seniority teachers’ 

beliefs, high-interaction practices, low-interaction practices, resources and attitudes. Compared with the teachers in 

the entry stage, the teachers in the appropriation and invention stage tended to have higher mean scores on all of 

dependent measures. It is not surprising that the teachers in higher stages of instructional evolution seemed to hold 

more positive belief and attitude toward technology and to have more resources to support technology-based 

instruction. In order to examine the effects of computer-based instructional evolution and teaching seniority on 

beliefs, high-interaction practices, low-interaction practices, resources and attitudes, a 5 (stages) × 5 (levels of 

teaching experience or seniority) MANOVA was employed. As Table 4 shows, there were significant differences 

regarding teachers’ beliefs in, attitudes toward, and practices of computer-based instruction. For instance, teachers in 

the “adaptation” and “appropriation” stages tended to have more positive beliefs about and attitudes toward 

computer-based instruction than those just in the “entry” stage; teachers in the “entry” stage also tended to perform 

low-interaction computer-based activities than those in the “appropriation” and “invention” stages; the latter two 

groups were still behind those in the “invention” stage as regards the degree of interactivity of their computer-based 

learning practices. The results imply few things: (1) teachers in the later stages (adaptation, appreciation, and 

invention) of computer-based instructional evolution held more positive beliefs and attitudes, and practiced more 

computer-based instructions in classroom; (2) teachers with positive beliefs and attitudes possibly moved to the later 

stages of computer-based instructional evolution and intended to practice computer-based instructions; (3) the 

successful teaching experiences in computer-based instructional evolution could promote teachers’ positive beliefs 

and attitudes, and encourage them to practiced more computer-based instructions in classroom. Therefore, teachers 

with different computer-based instructional evolution could be due to their different beliefs and attitudes; also, 

experiences of computer-based teaching practices could feedback to teachers’ beliefs, attitudes, and computer-based 

instructional evolution. 

As we can see in Table 4, teachers’ stage of instructional evolution and degree of teaching seniority had a significant 

impact on the amount of technical and personnel resources available at their schools. Since the interaction between 

123

teachers’ instructional evolution and seniority had reached a significant level, an ANOVA simple-effect analysis was 

conducted (see Table 5 and Figure 2). The results showed that teachers in the “entry” stage whose seniority was 6-10 

years reported that they had less technical and personnel resources available in their school than “entry” stage 

teachers whose seniority who were more than 21 years; these same teachers with 6-10 years of seniority in the 

“entry” stage also reported, very predictably, that they had less technical and personnel resources available in school 

than teachers with 6-10 years of seniority in the “appropriation” and “invention” stages. This means that teachers 

who hold computer-instruction skills and pedagogies could move to the later stage of computer-based instructional 

evolution if there are technical and personnel resources available in school. 

Table 3. Stage & Seniority vs. Beliefs, Practice, Resources, and Attitude 

Condition Beliefs Resources High practice Low practice Attitudes 

Mean SD Mean SD Mean SD Mean SD Mean SD 

< 5 years 

Entry 3.51 0.52 3.23 0.84 2.86 0.92 3.66 0.71 3.32 0.81 

Adoption 3.63 0.44 3.17 0.59 2.88 0.78 3.62 0.61 3.65 0.69 

Adaptation 3.70 0.49 3.20 0.67 2.92 0.87 3.85 0.56 3.86 0.62 

Appropriation 3.94 0.45 3.39 0.65 3.10 0.92 3.99 0.51 4.00 0.56 

Invention 3.65 0.40 3.23 0.79 3.61 0.66 3.78 0.76 3.43 0.83 

6-10 years 

Entry 3.29 0.52 2.77 0.60 2.52 0.81 3.42 0.79 3.49 0.74 

Adoption 3.64 0.52 3.20 0.63 2.95 0.95 3.73 0.51 3.71 0.52 

Adaptation 3.85 0.45 3.26 0.66 2.95 0.93 3.81 0.63 3.93 0.59 

Appropriation 3.99 0.45 3.61 0.59 3.32 0.75 4.01 0.71 4.09 0.60 

Invention 3.98 0.40 3.79 0.54 3.71 0.91 4.10 0.55 3.36 1.00 

11-15 years 

Entry 3.46 0.57 3.36 0.64 2.71 0.82 3.25 0.47 3.58 0.67 

Adoption 3.63 0.44 3.10 0.60 2.90 0.58 3.33 0.62 3.56 0.64 

Adaptation 3.74 0.51 3.20 0.72 2.86 0.86 3.76 0.57 3.62 0.68 


Invention 3.47 0.60 2.82 0.76 2.39 0.81 3.32 0.70 3.95 0.36 

16-20 years 

Entry 3.19 0.58 3.09 0.53 2.25 0.74 3.03 0.62 3.33 0.71 

Adoption 3.62 0.44 3.21 0.47 2.71 0.99 3.56 0.55 3.62 0.49 

Adaptation 3.69 0.41 3.37 0.65 2.65 0.55 3.58 0.72 3.56 0.63 


Invention 3.78 0.57 3.38 0.48 2.81 1.07 2.94 1.23 2.83 0.33 

> 21 years 

Entry 3.53 0.52 3.43 0.46 3.00 0.96 3.32 0.75 3.13 0.59 

Adoption 3.56 0.47 3.30 0.59 2.76 0.59 3.35 0.62 3.27 0.62 

Adaptation 3.67 0.58 3.42 0.70 2.76 0.80 3.49 0.74 3.39 0.71 


Invention 4.07 0.12 3.50 0.43 3.58 1.28 4.17 0.52 4.33 0.33 

Total 

< 5 years 3.70 0.48 3.23 0.68 2.98 0.87 3.79 0.61 3.74 0.69 

6-10 years 3.76 0.52 3.29 0.68 3.03 0.92 3.80 0.67 3.79 0.68 

11-15 years 3.65 0.51 3.18 0.65 2.87 0.77 3.49 0.63 3.64 0.64 

16-20 years 3.65 0.53 3.24 0.56 2.75 0.79 3.49 0.70 3.52 0.63 

> 21 years 3.65 0.51 3.42 0.60 2.82 0.83 3.46 0.71 3.40 0.66 

Total 

Entry 3.42 0.54 3.17 0.69 2.71 0.87 3.39 0.69 3.39 0.72 

Adoption 3.62 0.45 3.18 0.58 2.87 0.76 3.53 0.60 3.58 0.63 

Adaptation 3.73 0.49 3.26 0.68 2.88 0.85 3.76 0.62 3.75 0.66 


Invention 3.76 0.47 3.37 0.73 3.37 0.95 3.75 0.79 3.50 0.83 

124

Table 4. Summary of MANOVA Results 

Condition Beliefs Resources High practice Low practice Attitude 

F E.S. F E.S. F E.S. F E.S. F E.S. 

Main effect 

Stages 11.86 *** 0.29 1.76 0.11 4.29 ** 0.17 6.51 *** 0.21 6.00 *** 0.20 

Post Hoc (1

Regression models indicating relationships 

The questions in the third section of the questionnaire were grouped by dependent variable (high- and lowinteraction 

practices) and independent variable (beliefs, attitudes, and resources). The means of the responses were 

then used to represent each variable, since this is one of the most-used parameters to represent a group of values 

(Moore & Benbasat, 1991; Holcombe, 2000). After calculating these means, an analysis of the correlation between 

variables was done. As we can see in Table 6, there were significant correlations among teachers’ beliefs, attitude 

toward computers, available resources, and teaching practices (including high- and low- interaction practices). 

Beliefs show the greatest correlation with the dependent variables (high- and low-interaction practices); therefore 

beliefs will have a greater impact in the regression models. 

Table 6. Correlation Matrix for the Five Teacher Factors (N = 613) 

Pearson Correlation Beliefs H-Practice L- Practice Attitude Resources 

Beliefs 1 

H-Practice 0.31** 1 

L- Practice 0.48** 0.44** 1 

Attitude 0.33** 0.09* 0.32** 1 

Resources 0.27** 0.21** 0.16** 0.05 1 

**Correlation is significant at the 0.01 level (2-tailed). 

*Correlation is significant at the 0.05 level (2-tailed). 

As we can see in Table 7, these coefficients of the multiple linear regressions were grouped into two categories: (1) 

the dependent variable is high-interaction practices; (2) the dependent variable is low-interaction practices. In each 

category six regression models, including one group with all the subjects and five groups with the subjects 

distributed into five stages, were calculated in order to compare the different contributions of variables (beliefs, 

attitudes toward computers, and available resources). Teachers’ beliefs and available resources turned out to be the 

major predictors of their high-interaction practices. Besides, all of the t values reached a significant level (0.05), 

which means that independent variables (beliefs and available resources) contributed significantly to the prediction 

of the dependent variable (high-interaction practices). The same procedure was used to analyze the subjects in the 

five stages of computer-based instructional evolution. As we can see in Table 7, beliefs are the greatest predictor of 

high-interaction practices except at the “entry” stage; here teachers’ beliefs and available resources are the best 

predictors of this variable. 

By contrast, beliefs and attitudes contribute significantly to the prediction of low-interaction practices. As we can 

see, beliefs are the greatest predictor for these practices except at the “adaptation” stage. In this stage, teachers’ 

beliefs and attitudes best predict that teachers will engage in low-interaction activities when they integrate computerbased 

technology into their classrooms. In conclusion, teachers’ beliefs are the most reliable predictors of their 

computer-based instructional practices. Similar findings can be found in many studies (Czerniak & Lumpe, 1996; 

Niederhauser & Stoddart, 2001; Ruthven, et al., 2004; Sandholtz, Ringstaff & Dwyer, 1996 ). 

Total 

(N=613) 

Entry 

(N=89) 

Table 7. Summary of Regression Models 

High-Interaction Practice 

Variables Beta S.E. t (p) R 

Constant 0.63 0.26 2.41(.016) 

Beliefs 0.47 0.07 6.97(.000) 

.34 

Resources 0.18 0.05 3.37(.001) 

Constant 0.29 0.56 0.51(.061) 

Resources 0.34 0.14 2.50(.015) 

.44 

Beliefs 0.39 0.18 2.19(.031) 

Constant 

Beliefs 

1.82 

0.29 

0.51 

0.14 

3.55(.001) 

2.04(.043) 

.17 

Constant 

Beliefs 

1.41 

0.39 

0.42 

0.11 

3.33(.001) 

3.49(.001) 

.23 

Constant 

Beliefs 

0.31 

0.73 

0.61 

0.16 

0.50(.615) 

4.71(.000) 

.41 

Adoption 

(N=141) 

Adaptation 

(N=229) 

Appropriation 

(N=111) 

Invention Constant 0.21 1.09 0.19(.850) .42 

126

(N=43) Beliefs 0.84 0.29 2.93(.006) 

Low-Interaction Practice 

Variables Beta S.E. t (p) R 

Total 

(N=613) 

Entry 

(N=89) 

Adoption 

(N=141) 

Adaptation 

(N=229) 

Appropriation 

(N=111) 

Invention 

(N=43) 

Conclusion 

Constant 0.97 0.19 5.19(.000) 

Beliefs 0.56 0.05 11.46(.000) 

Attitude 0.17 0.04 4.73(.000) 

Constant 1.85 0.45 4.14(.000) 

Beliefs 0.45 0.13 3.45(.001) 

Constant 1.38 0.37 3.74(.000) 

Beliefs 0.59 0.10 5.88(.000) 

Constant 1.000 0.30 3.34(.001) 

Beliefs 0.48 0.08 6.23(.000) 

Attitude 0.26 0.06 4.55(.000) 

Constant 0.65 0.47 1.37(.172) 

Beliefs 0.48 0.12 4.00(.000) 

Attitude 0.34 0.09 3.69(.000) 

Constant 0.15 0.83 0.18(.855) 

Beliefs 0.96 0.22 4.39(.000) 

The purpose of this study was to identify the factors which influence teachers’ computer-based instructional 

practices. After conducting a survey, we found teachers’ beliefs, attitudes toward computers, and available school 

resources to be the factors that most affected their computer-based teaching practices. The contribution of each factor 

to teachers’ computer-based instruction is different for different stages of instructional evolution. Teachers’ beliefs 

and available resources are the major predictors of their high-interaction practices; beliefs and attitudes contribute 

significantly to the prediction of low-interaction practices. Overall, teachers’ belief is the biggest predictor of the 

way in which they practice computer-based instruction. The next question is then: what affects teachers’ beliefs, 

attitudes, and available resources in schools? The findings indicated that teachers in the “adaptation” and 

“appropriation” stages have more positive beliefs and attitudes regarding computer- based instruction than those still 

in the “entry” stage, while their teaching seniority has no significant influence on beliefs, attitudes and available 

resources. The interaction between teachers’ instructional evolution and seniority reached a significant level when 

measured against the dependent variable of available resources in school. Teachers with different seniorities and at 

different stages of instructional evolution regarded “available resources in school” as a crucial factor. Teachers who 

had taught for 6 to 10 years at the “entry” stage reported that they had less technical and personnel resources 

available in school than the teachers in the “appropriation” and “invention” stages. In other words, after teachers has 

taught for 6 to 10 years they could move to the “appropriation” and “invention” stages as long as they felt there were 

plentiful personnel and technical resources available in school. 

The regression analysis shows teachers’ belief is the best predictor of their computer-based instruction practices. A 

useful focus of future research would be the issue of how to create the learning environments in which teachers 

might best learn the necessary computer-instruction skills and pedagogies, and might best develop more positive 

beliefs and attitudes with regard to computer-based learning. Perhaps training programs must allow especially 

“entry”-stage teachers to visit expert teachers’ classes, in order to see how they use computer technology to interact 

in intensive and exciting ways with their students. Such “instruction” is after all the best way to change teachers’ 

beliefs about and attitudes toward computers. School culture is another factor to affect teachers’ beliefs. The 

implementation of computer-based instruction at school is consistent with the existing beliefs and practices of school 

members including teachers and administrators (Zhao, Pugh, Sheldon & Byers, 2002). How to change school culture 

for classroom technology innovations is an important issue and educational policy makers need to find an 

encouraging way to promote it. 

From the results, it is strongly suggested that teachers could move to the later stage of computer-based instructional 

evolution if there were technical and personnel resources available in school and teachers hold positive beliefs about 

using computers in classroom. Therefore, if educational policy is to integrate computers into teaching and learning, 

the school managers need to provide necessary technical and personnel supports for teachers’ computer-based 

instruction, encourage teachers to share their experiences in innovative instruction, and shape teachers’ development 

.51 

.35 

.45 

.53 

.56 

.57 

127

of beliefs about integrating computers into teaching and learning. More instructional examples and design principles 

are needed to facilitate teachers to create efficient instructions that enable to promote meaningful learning. 


This research project was funded by the National Science Council of the Republic of China under contract no. NSC 

92-2511-S-003-039 and NSC 95-2524-S-003-012. The author gratefully acknowledges the collaboration of her 

colleagues: Tai-Yih Tso and Chang Yung-Ta. 

References 

Angeli, C., & Vanlanides, N. (2005). Preservice elementary teachers as information and communication technology 

designers: an instructional systems design model based on an expanded view of pedagogical content knowledge. 

Journal of Computer Assisted Learning, 21, 292-302. 

Beaudoin, M. F. (1990). The instructor’s changing role in distance education. The American Journal of Distance 

Education 4, 21-29. 

Becker, J. (2000a). Findings from the teaching, learning, and computing survey: Is larry cuban right? Educational 

Policy Analysis Archives, 8, retrieved October 15, 2007, from http://www.crito.uci.edu/tlc/findings/ccsso.pdf. 

Becker, J. (2000b). Who's wired and who's not: Children's access to and use of computer technology. The Future of 

Children, 10 (2), 44-75. 

Borphy, J., & Good, T. L.(1986)Teacher behavior and student achievement. In M. C. Wittrock (Ed.), Handbook 

of Research on Teaching, New York, NY: Macmillan. 

Chiero, R. T. (1997). Teachers' perspectives on factors that affect computer use. Journal of Research on Computing 

in Education, 30 (2), 133-126. 

Comber, C., Colley, A., Hargreaves, D. J., & Dorn, L. (1997). The effects of age, gender and computer experience 

upon computer attitude. Educational Research, 39 (2), 123-133. 

Czerniak, C. M., & Lumpe, A. T. (1996). Relationship between teacher beliefs and science education reform. 

Journal of Science Teacher Education, 7 (4), 247-266. 

Ercan, K., & Ozdemir, D. (2006). The relationship between educational ideologies and technology acceptance in preservice 

teachers. Educational Technology & Society, 9 (2), 152-165. 

Fetters, M.K., & Vellom, P. (2001). Linking schools and universities in partnership for science teacher preparation. 

In Lavoie, D. R. & Roth, W. M. (Eds.), Models of science teacher preparation: Theory into practice, The 

Netherlands: Kluwer Academic Publishers, 97-88. 

Gardner, D. G., Discenza, R., & Dukes, R. L. (1993). The measurement of computer attitudes: An empirical 

comparison of available scales. Journal of Educational Computing Research, 4, 487-507. 

Hargrave, C. P., & Thompson, A. D. (2001). TEAMS: A science learning and teaching apprenticeship. In Lavoie, D. 

R. & Roth, W. M. (Eds.), Models of science teacher preparation: Theory into practice, The Netherlands: Kluwer 

Academic Publishers, 11-30. 

Holcombe, M.C. (2000). Factors influencing teacher acceptance of the Internet as a teaching tool: a study of Texas 

schools receiving a TIF or a TIE Grant, Unpublished dissertation, Baylor University, School of Education. USA. 

128

Hsu, Y.-S., Cheng ,Y.-J., & Chiou, G.-F. (2003). Internet uses in the school: A case study of the Internet adoption in 

a senior high school. Innovations in Education and Teaching International, 40 (4), 356-368. 

Jennings, S. E., & Onwuegbuzie, A. J. (2001). Computer attitudes as a function of age, gender, math attitude, and 

developmental status. Journal of Educational Computing Research, 25 (4), 367-384. 

Jonassen, D. H. (1996). Computers in the classroom: Mindtools for critical thinking, Englewood Cliffs, NJ: Prentice- 

Hall. 

Kay, R. (1989). A Practical and Theoretical Approach to Assessing Computer Attitudes: The Computer Attitude 

Measure (CAM). Journal of Educational Computing Research, 3, 456-463. 

Lavoie, D. R. (2001). New technologies and science teacher preparation. In Lavoie, D. R. & Roth, W. M. (Eds.), 

Models of science teacher preparation: Theory into practice, The Netherlands: Kluwer Academic, 163-176. 

Loyd B. H., & Gressard, C. (1984). Reliability and factorial Validity of Computer Attitude Scales. Educational and 

Psychological Measurement, 2, 501-505. 

McKenzie, B. K., Mims, N. G., Davidson, T. J., Hurt, J., & Clay, M. N. (1996). The design and development of a 

distance education training model. Paper presented in Society for Information Technology and Teacher Education 

annual conference, March 13-16, 1996, Phoenix, Arizona, USA. 

Ministry of Education in Taiwan (1999). National standards of 1-9 science and technology curriculum, Taipei, 

Taiwan: Ministry of Education in Taiwan. 

Mooij, T., & Smeets, E. (2001). Modelling and supporting ICT implementation in secondary schools. Computers & 

Education, 36, 265-281. 

Moore, G.C., & Benbasat, I. (1991). Developemnt of an instrument to measure the perceptions of adopting an 

information technology innovation. Information Systems Research, 2 (3), 192-222. 

Niederhauser, D. S., & Stoddart, T. (2001). Teachers’ instructional perspectives and use of educational software. 

Teaching and Teacher Education, 17, 15-31. 

OTA, O. T. A. (1995). Teachers and technology: Making the connection, Washington, DC: Government Printing 

Office. 

Pelgrum, W. J. (2001). Obstacles to the integration of ict in education: Result from a world wide educational 

assessment. Computers & Education, 37, 163-178. 

Ravitz, J., Becker, H., & Wong, Y. (2000). Constructivist-compatible beliefs and practices among U.S. Teachers, 

Irvine,CA: Center for Research on Information Technology and Organizations. 

Rogers, P. L. (2000). Barriers to adopting emerging technologies in education. Journal of Educational Computing 

Research, 22 (4), 455-472. 

Russell, M., Bebell, D., O’Dwyer, L., & O’Connor, K. (2003). Examining teacher technology use: Implication for 

pre-service and in-service teacher preparation. Journal of Teacher Education, 54 (4), 297-310. 

Ruthven, K., Hennessy, S., & Brindley, S. (2004). Teacher representations of the successful use of computer-based 

tools and resources in secondary-school English, Mathematics and Science. Teaching & Teacher Education, 20 (3), 

259-275. 

Sandholtz, J. H., Ringstaff, C., & Dwyer, D. C. (1996). The evolution of instruction in technology: rich classroom. In 

J. H. Sandholtz, C. Ringstaff, & D. C. Dwyer (Eds.), Teaching with technology: Creating student-centered 

classrooms, New York and London: Teachers College Press, 33-54. 

129

Sinko, M., & Lehtinen, E. (1999). The challenge of ICT in Finnish education, Jyväskylä, Finland: Atena. 

Smeets, E., Mooij, T., Bamps, H., Bartolomé, A., Lowyck, J., Redmond, D., & Steffens, K. (1999). The impact of 

Information and Communication Technology on the teacher, Nijmegen, The Netherlands: KU/ITS, retrieved October 

15, 2007, from http://webdoc.ubn.kun.nl/anon/i/impaofina.pdf. 

Teo, H. H., & Wei, K. K. (2001). Effective use of computer aided instruction in secondary schools: A causal model 

of institutional factors and teachers’ role. Journal of Educational Computing Research, 25 (4), 285-315. 

Thatch, E. C., & Murphy, K. L. (1995). Competencies for Distance Education Professionals. Educational 

Technology Research and Development, 43, 57-72. 

van Braak, J. (2001). Individual characteristics influencing teachers’ class use of computer. Journal of Educational 

Computing Research, 25 (2), 141-157. 

Verduim, J. R., & Clark, T. A. (1991). Distance Education: The Foundations of Effective Practice, San Francisco, 

CA: Jossey-bass. 

Windschitl, M., & Sahl, K. (2002). Tracing teachers’ use of technology in a laptop computer school: The interplay of 

teacher beliefs, social dynamics, and institutional culture. American Educational Research Journal, 39, 165-205. 

Winn, W. (1993). Instructional Design and Situated Learning: Paradox or Partnership. Educational Technology, 33 

(3), 16-21. 

Woodrow, J. J. (1994). The development of computer-related attitude of secondary students. Journal of Research on 

Computing in Education, 7 (2), 165-187. 

Yildirim, S. (2000). Effects of an Educational Computing Course on Pre-service and In-service Teachers: A 

Discussion and Analysis of Attitude and Use. Journal of Research on Computing in Education, 4, 479-496. 

Zhao, Y., & Frank, K. A. (2003). Factors affecting technology uses in schools: An ecological perspective. American 

Educational Research Journal, 40 (4), 807-840. 

Zhao, Y., Pugh, K., Sheldon, S., & Byers, J. L. (2002). Conditions for classroom technology innovations. Teachers 

College Record, 104 (3), 482-515. 

130

Deryakulu, D., & Olkun, S. (2007). Analysis of Computer Teachers’ Online Discussion Forum Messages about their 

Occupational Problems. Educational Technology & Society, 10 (4), 131-142. 

Analysis of Computer Teachers’ Online Discussion Forum Messages about 

their Occupational Problems 

Deniz Deryakulu 

Department of Computer and Instructional Technologies Education, Ankara University, Turkey 

Tel: +90 (312) 363 3350 Ext: 3203 // Fax: +90 (312) 363 6145 // deryakul@education.ankara.edu.tr 

Sinan Olkun 

Department of Elementary Education, Ankara University, Turkey 

Tel: +90 (312) 363 3350 Ext: 5111 // Fax: +90 (312) 363 6145 // sinanolkun@gmail.com 

ABSTRACT 

This study, using content analysis technique, examined the types of job-related problems that the Turkish 

computer teachers experienced and the types of social support provided by reciprocal discussions in an online 

forum. Results indicated that role conflict, inadequate teacher induction policies, lack of required technological 

infrastructure and technical support, and the status of computer subject in school curriculum were the most 

frequently mentioned problems. In addition, 87.9% of the messages were identified as providing emotional 

support, while 3.1% messages were identified as providing instrumental support. It is concluded that content 

analysis technique provides an invaluable tool to understand the nature of communication and social interaction 

patterns among users in online environments. CMC in education should not only be considered to be a tool for 

content delivery and instructional interaction, but also a feedback mechanism and a platform for professional 

support, as well as an informal learning environment. 

Keywords 

Teacher stress, Social support, Online discussion forums, Computer-mediated communication 

Introduction 

Many studies have examined job-stress, burnout and job dissatisfaction among teachers (Abel & Sewell, 1999; 

Farber, 1984; Friedman, 1991; van Dick & Wagner, 2001). These studies demonstrated that teachers often 

experience a great deal of stress when there was an imbalance between the demands of the job environment and their 

response capability. Wolpin, Burke, and Greenglass (1991) found that the negative work setting characteristics 

resulted in greater work stressors that in turn were associated with increased teacher burnout thus resulted in 

decreased job satisfaction. Some other studies revealed that when these negative working conditions merge with poor 

staff communication and lack of social support that might result in increased teacher stress and burnout (Black, 2003; 

Brissie, Hoover-Dempsey, & Bassler, 1988; Burke & Greenglass, 1993). 

Although there is a huge body of research on teacher stress and burnout, few studies have specifically dealt with 

computer teachers (see Deryakulu, 2005, 2006). Most of these studies concerning teacher stress and burnout used 

self-report as a source of data. There are extensive critiques related to the use of self-report measures as having 

significant shortcomings in assessing sources, types and levels of job-stress and burnout (see Guglielmi & Tatrow, 

1998). In addition, self-report questionnaires are susceptible to the negative effects of social desirability (see Evers, 

Brouwers, & Tomic, 2002). Therefore, we decided to use a different data source to examine the sources of jobstresses 

for computer teachers in order to identify potential barriers to effective and efficient computer education. 

The present study aims at analyzing the content of messages posted by Turkish computer teachers in an online 

asynchronous discussion forum about their occupational problems. We believe the examination of the content of 

messages in this specific online discussion forum would help us portray the major stress-inducing problems of 

computer teachers, as well as the types of social support provided by reciprocal online discussions. Furthermore, 

examining the kind of data derived from an open, spontaneous online discussion forum (not initiated by an external 

researcher) could help us identify new factors, which otherwise could not be obtained by self-reported measures. 

Background 

In the following sections, we first provide an overview on the nature of computer education in Turkey and review the 

existing research findings concerning stress and burnout in computer teachers. Second, we briefly introduce the 






131

concept of social support. Lastly, we present the potential use of computer-mediated communication tools for 

providing online social support. 

The Nature of Computer Education in the Turkish Educational System 

In 1998, The Turkish Ministry of National Education (MNE) received a loan from the World Bank for the Basic 

Education Program (BEP). The primary aims of the BEP are to expand the scope of basic education and to improve 

the quality of education. The MNE also set such aims as to ensure that each student and teacher become computer 

literate, to integrate ICT into school curriculum, and to establish computer laboratories in schools (MEB, 2004). In 

this context, computer as an elective subject was added to the elementary school curriculum in 1998 as 1 or 2 hours 

per week for grades 4 through 8, and later was added to the academic high school curriculum in 2000 for grades 9 

and 10. The primary aim of this course is to increase the number of computer literate students to facilitate and 

accelerate the diffusion of ICT usage across school curriculum. In 2005, the MNE allowed students to take computer 

subjects as electives from the first to the eighth grade. However, total teaching time is restricted to 1 hour per week. 

Turkey still has difficulties in widening computer literacy among students and integrating ICT into school 

curriculum. Many students still do not have access to computers in the teaching-learning processes of other subjects 

such as science and mathematics in public schools (Olkun, Altun, & Smith, 2005). Thus, the only opportunity for 

many students to have access to computers (especially in high-poverty areas) might be elective computer subjects. 

However, there are 41,091 public schools housing 12.7 million students in Turkey, while the number of computer 

teachers who graduated from an accredited computer teacher-training program is only 10% of the number of schools 

(MEB, 2004, 2005b). According to these statistics, majority of the schools with computers have no computer teacher 

specifically trained for teaching computers. The MNE has been trying to solve computer teacher shortage by 

employing one computer teacher for two or more schools, employing computer coordinators as computer teachers or 

allowing schools to make contract with individuals who have the relatively appropriate background and skills. In 

short, computer teachers may come from different disciplines in Turkish educational system. This causes other 

problems especially dissatisfaction among accredited computer teachers in terms of employee rights and 

responsibilities. For example, computer engineers employed as computer teachers enjoy better payment and seniority 

advantages while accredited computer teachers do not. 

Recently, Deryakulu (2005) examined the levels of burnout in Turkish computer teachers using the Maslach Burnout 

Inventory-Turkish Version. Because there are no norms for this inventory, by using percentage values, she exposed 

that the surveyed computer teachers displayed fewer symptoms of depersonalization, relatively moderate symptoms 

of emotional exhaustion and reduced personal accomplishment. In this study, using an open-ended form, lack of 

technical support, lack of student interest, and large class sizes were found to be the foremost stress-inducing 

problems that the computer teachers experienced. 

Deryakulu (2006) also examined the factors predicting burnout in Turkish computer teachers. She found two 

significant predictors of emotional exhaustion and depersonalization. These were the types of problems they 

encountered while teaching computers under a heavy teaching load. These findings suggest that the more the 

teachers suffered from job-stress, the more they showed symptoms of burnout. As is well known, job-stress and 

burnout may cause decline in teachers’ job performance and also may be detrimental to the physical and 

psychological health of teachers. Therefore, we have to think about the ways to reduce teachers’ job-stress to prevent 

our teachers and students from its harmful effects. One of the effective ways to reduce job-stress is to provide social 

support to those who are stressed. 

The Concept of Social Support 

Social support has been identified as a resource provided by another person that enables individuals to cope with 

stress (Russell, Altmaier, & Van Velzen, 1987). There are many forms of social support. Beehr and Glazer (2001) 

have mentioned two main social support categories: (a) structural and (b) functional. According to Cohen and Wills 

(1985), structural support means that a relationship with one or more others exists. Structural support does not 

indicate however what other people do or what functions they perform for the focal person. Functional support, on 

132

the other hand, implies that the supportive people are performing some functions for the focal person and this kind of 

support must perform at least one of these two functions: (a) emotional or (b) instrumental (Beehr & Glazer, 2001). 

Emotional support means that the supportive people make the focal person feel emotionally better in a number of 

ways while instrumental support is the kind of help or assistance from other people that tangibly helps focal person 

to solve a problem or get a task done (Beehr & Glazer, 2001). Communicating with the focal person and providing 

him/her with feedback, praise or approval are the examples of emotional support. Instrumental support, on the other 

hand, includes such support as doing physical or mental labor or providing informational or financial resources that 

make it easier for the focal person to solve his/her problems or complete a stress-inducing task (Beehr & Glazer, 

2001). 

In school settings, support from a teacher’s colleagues has been identified as preventive and remedial mechanisms 

for job-stress and burnout as well as an aid in coping with the job demands (see Brissie et al., 1988). Therefore, to 

widen the size of social interaction network of a teacher may increase the probability of available social support. 

Computer-mediated communication (CMC) tools can contribute to widening the size of social interaction network 

that can function as an alternative channel for giving and receiving social support. 

Computer-Mediated Communication and Online Social Support 

Computer-mediated communication refers to both synchronous and asynchronous modes of communication between 

individuals and among groups via networked computers (Naidu & Järvelä, 2006). Electronic mail, listservs, 

newsgroups and online discussion forums are just a few examples of CMC tools. Among these, online discussion 

forums provide an open electronic environment that allows the member (user) to post a message on a specific topic 

for others to read. Other members can respond to this message asynchronously. This usually leads to the emergence 

of threads, where a number of participants provide responses and counter-responses to an original posting, thus 

forming a dialogue among contributors. Participation to an open online discussion forum is voluntary. Therefore, it 

can be thought that participants are self-motivated, purposeful, and willing to express their experiences, emotions 

and thoughts, and listen to others’ concerns related to the issue(s) being discussed. In this process, participants must 

engage in a “conversation” to express themselves to obtain and provide social support. Burleson and Goldsmith 

(1998, p.260) described conversation as “a medium in which a distressed person can express, elaborate, and clarify 

relevant thoughts and feelings.” Obviously, such kind of personal participation requires friendly, safe, nonthreatening, 

and comfortable environments. Caplan and Turner (2007) suggest that establishing such an environment 

may be easier and more effective if the conversation is computer-mediated. According to Walther and Parks (2002), 

the Internet is a successful medium for social support. Recent studies indicated that both emotional and instrumental 

support can be found in online communication (Eastin & LaRose, 2005; Weisskirch & Milburn, 2003). In addition, 

there has been strong evidence that frequent use of online communication increased perceived social support, which 

in turn reduced users’ stress levels (Wright, 2000). Therefore, we tried to find answers to the following questions: 

1. What types of job-related problems are reflected in online discussion forums by computer teachers? 

2. What types of social support do computer teachers provide to one another within an online discussion forum? 

Method 

Sample 

This study examined the content of messages posted by Turkish computer teachers to a specific online asynchronous 

discussion forum –entitled “Unfairness that Computer Teachers Encounter” 

(http://forum.memurlar.net/topic.aspx?id=58164). This specific forum was voluntarily opened by computer teachers 

on September 27, 2005, and turned out to be a very active threaded-discussion forum for these civil servants. Its 

main aims were to provide a common ground for computer teachers to share and discuss their occupational problems 

with their colleagues and to let the policy and decision makers of the Ministry of National Education know about 

these problems. Participation to the forum was also voluntary. In other words, we did not do any effort to get 

teachers involved in the discussions. There were 128 anonymous participants. We collected all forum postings in the 

period of 12 weeks. A total of 543 messages were analyzed. 

133

Procedure 

Content analysis procedures were applied to the messages. Content analysis is “a research technique for the 

objective, systematic, quantitative description of the manifest content of communication” (Berelson, 1952, p.18). 

Quantitative description process includes segmenting communication content into units, assigning each unit to a 

category, and providing tallies for each category (Rourke & Anderson, 2004). Hara, Bonk, and Angeli (2000) note 

that there is no standard scheme for analyzing the computer-mediated communication, but they suggest gathering 

quantitative information about the number and type of messages and qualitative information about the content of 

messages. We used the individual text-based message as the unit of analysis (segment) that has significant 

advantages such as being objectively identifiable for raters (see Rourke, Anderson, Garrison, & Archer, 2001). Since 

the forum message is a fixed unit which is determined by horizontal lines representing the start and end of sentences 

in the communication transcript, a separate segmentation procedure was not applied. Thus, a segmentation reliability 

coefficient was not calculated. 

The variables investigated in this study were the types of job-related problems, which computer teachers mentioned 

in their messages and the types of social support which provided by teachers to one another via reciprocal 

discussions. When we were identifying the computer teachers’ job-related problems we used an inductive approach 

since we did not have proper pre-determined problem categories. That is, coding categories were derived from the 

data set by the authors. While we were classifying the types of social support, we on the other hand, used a deductive 

approach. In other words, a well-founded pre-existing coding schema (i.e., emotional and instrumental support; see 

Beehr and Glazer, 2001) was used. 

After multiple readings of the messages the authors derived tentative problem categories mentioned by the computer 

teachers. The first author trained a research assistant for coding. They independently assigned each unit to a 

category, and provided tallies for each category in order to quantify the data. During the coding, however, initial 

tentative coding categories were modified in accordance with the categories emerging from the data. As 

recommended by De Wever, Schellens, Valcke, and Van Keer (2006), we used more than one method for calculating 

inter-rater reliability coefficients in order to present more evidence about the reliability of classification. These 

coefficients are reported in Table 1. Differences in classification between the two raters were resolved by discussion. 

Results 

Problem Analysis 

Table 1. The inter-rater reliability coefficients 

Method Classification 

Problem Support 

Cohen’s kappa 0.98 0.92 

Kendall’s tau-b 0.98 0.93 

Spearman’s rho 0.98 0.94 

The content analysis revealed that almost half of the messages (49.5%; f=269) contained expressions of problems. 

Since some of these messages comprised of more than one type of problem a total of 375 problem expressions were 

identified. These problems were grouped under 12 categories as depicted in Table 2. 

Table 2. The types of job-related problems of Turkish computer teachers 

Problem Categories f % 

1- Role Conflict 96 25.6 

2- Inadequate Teacher Induction Policies 95 25.3 

3- Lack of Required Technological Infrastructure and Technical Support 77 20.5 

4- The Status of Computer Subject in School Curriculum 42 11.2 

5- Lack of Appreciation and Positive Feedback from Colleagues 19 5.1 

6- Unsupportive Administrators 9 2.4 

7- Rapidly Changing Nature of Content Knowledge in Computer Education 8 2.1 

8- Lack of Cohesive Computer Curriculum 8 2.1 

134

9- Insufficiency of Pre-Service Teacher Training Programs 7 1.9 

10- Large Class Sizes 7 1.9 

11- Indifferent Students 4 1.1 

12- Inadequate Supervision and Inspection 3 0.8 

Total 375 100 

As can be seen in Table 2, many of the problems stated by the Turkish computer teachers are mainly related to 

educational policy and organizational factors. The three most common job-related problems the Turkish computer 

teachers faced were role conflict, inadequate teacher induction policies, and lack of required technological 

infrastructure and technical support. As stated earlier, these problem categories were derived from the messages 

posted by computer teachers. For example, one computer teacher wrote the following about his/her experience of 

“role conflict”: 

Now I terribly regret that I became a computer teacher. It has only been a month since I started to 

work but it seems like ten years. One thing that really upsets and makes me angry is to be looked at as 

“a handy-man”, “a repairman”. Besides the computers that need to be repaired at school, I also am 

frequently asked to fix the computers at teachers’ boarding house. Honestly, I really want to leave my 

profession. We are teachers, for goodness sake not repairmen... (Message 206) 

As shown in the excerpt, when teachers are expected to perform a demand that could be considered to be unrelated to 

their actual job description, stress is often the consequence. These kinds of demands often cause role conflict in 

teachers. Role conflict is defined as the simultaneous occurrence of two or more sets of inconsistent, expected role 

behaviors and has been found to be a major source of teacher stress in many different studies (see Cooper & 

Marshall, 1978; Kyriacou, 2001; Schwab & Iwanicki, 1982; Schwab, Jackson, & Schuler, 1986). Due to inadequate 

technical facilities and support services in most public schools, the computer teachers are expected to repair broken 

down computers or to clean computer labs in addition to their routine teaching responsibilities. Therefore, computer 

teachers’ roles and responsibilities should be more clearly described. 

Another computer teacher wrote the following complaint about the “inadequate teacher induction policies”: 

... Those who do not have relevant education are employed as computer teachers. As a result, those 

who graduated from universities after years of studying, sometimes ranks lower than computer 

coordinators. Who are these computer coordinators? They are teachers from other disciplines. 

Following a short-term in-service education, they are employed as computer teachers in the case of 

lack of accredited computer teachers. But now it seems there are plenty of those people around… I 

now would like to ask our Minister of National Education, why we, the computer teachers, are put in a 

position to compete with computer coordinators who in reality lack the real computer education. And 

why do you employ them at well-equipped schools, and leave repairs and fixing jobs to the properlyeducated 

computer teachers. (Message 192) 

As mentioned before, because of the shortage of accredited computer teachers, persons with different backgrounds 

such as engineers, programmers, classroom teachers or statisticians can be employed as computer teachers in the 

Turkish school system. Among them, only the accredited computer teachers have proper and higher level of 

education related to teaching computers. The literature suggests that higher level of education usually leads to higher 

career aspirations (Friedman, 1991). Depending on their higher career aspirations, accredited computer teachers 

might expect more respect of their expertise, and consequently might be relatively more sensitive or intolerant of the 

employment of out-of-field persons as computer teachers at public schools. Instead of employing out-of-field persons 

as computer teachers, existing computer teachers should be utilized more effectively. Namely, computer teachers 

should not be employed at schools which have no computers. 

A computer teacher wrote the following about the “lack of required technological infrastructure and technical 

support” preventing effective and efficient computer education in schools: 

...in my computer lab, seven of the computers have Windows 3.1, the other six have Windows 95. 

Although it is the year 2005, we use the first version of Windows operating system. …The keyboards 

135

and mice are out of order. I have to explain MS Word, MS Excel and other subjects by writing it on 

the blackboard. In some schools, some of our colleagues work with 486 DX computers. The system 

configurations of computers are so out of date that it is impossible to update to new OS and software. 

Moreover, in some schools there are computer teachers but no computers. (Message 319) 

Similar complaints could be found in the following quote by a computer teacher about “the status of computer 

subject in school curriculum” and the “lack of required technological infrastructure and technical support” in 

schools: 

Look friends; let’s raise our voices against the move toward lowering computer courses to one hour 

per week. I give computer courses at two different schools, one of which does not have a computer 

lab. I am trying to teach computers to the students most of whom have never set an eye on one. In the 

other school I work for, there are really out-dated machines that only come to life in 15 minutes after 

switching on. So the result is I spend half of one-hour-class to start the computers and to restore 

classroom discipline. How much can the students benefit from this? Can anybody have a guess? 

(Message 470) 

As stated earlier, the status of computer subject in the Turkish elementary school curriculum is elective and the total 

teaching time is merely one hour per week. Accordingly, the computer teachers frequently considered the status of 

this subject to be a restrictive factor decreasing the effectiveness and efficiency of their teaching practices. They 

consistently expressed that, because the computer subject was one hour per week, they were not able to provide 

enough hands-on practice for each student. Furthermore, because the computer subject was elective, the majority of 

students considered this subject to be unimportant. Similarly, Hendley, Stables, and Stables (1996) suggested that the 

subjects which occupy little time in the curriculum may be regarded as of low status by students. Therefore, the 

weekly course hours should be increased to provide enough hands-on practice for each student. Increasing the total 

teaching time may also help to improve students’ perceptions regarding the significance of this subject. 

Furthermore, without well-equipped computer labs and technical supports it is impossible for computer teachers to 

continue their classes properly. Studies suggest that poor working conditions including lack of educational supplies 

and inadequate resources for teaching may lead to job-stress in teachers (Abel & Sewell, 1999; Kyriacou, 2001). 

Besides, when lack of required technological infrastructure and technical support merge with the limited teaching 

time, teacher stress and ineffective computer education seem unavoidable. For an effective and efficient computer 

education in schools, technological infrastructure, hardware and software investments should not be one-shot 

investments. Instead, continuous technological renovation and technical support services should be provided. 

Following excerpt is a typical complaint about the “lack of appreciation and positive feedback from his/her 

colleagues” from a computer teacher: 

It is my third year in teaching. I have chosen willingly to become a computer teacher with full of ideas 

in the beginning. However all my enthusiasm has gone astray after the first year... For one thing, you 

are extremely overloaded with works. I have given computer courses to the teachers at every seminar 

session in my school. While the computer coordinators are paid a hell of a lot of money for doing such 

courses, I was even spared a simple “thank you.” I designed the school’s web site… which I believe 

could not to be done for less than $500 by somebody from the outside… I was criticized for not 

updating it regularly. I don’t want to be misunderstood on money matters, but I certainly think that 

one needs to be at least appreciated for the work one does. As a result, now my feet go backwards 

when I go to my classes. Does a working person look for his/her retirement on the third year, well I 

do... I do love teaching, but not under these conditions. It is high time that our complaints should now 

be heard and acknowledged by authorities. Reading similar complaints from our colleagues, we feel 

we are not alone, but these should not fall on deaf ears. (Message 118) 

Teachers usually need to receive positive feedback, praise or approval from their colleagues and support from 

administrators to cope with stressful job demands. Otherwise, they may feel that their work is not important enough 

to justify someone else’s attention and they are alone in trying to do their work (see Pines, 2002). Lack of feedback 

and support were identified as causing additional stress in teachers (see Brown & Nagel, 2004; Kyriacou, 2001). 

However, job-stress can be reduced by positive communication and supportive relationships among colleagues. In 

136

this context, CMC seems to open new doors for teachers who need to set up supportive communication networks 

with their colleagues. 

Furthermore, unsupportive administrators, the rapidly changing nature of content knowledge in the field of computer 

education, lack of cohesive computer curriculum, poor university preparation, large class sizes, indifferent students, 

and inadequate supervision and inspection were found to be relatively less frequently mentioned problems of the 

Turkish computer teachers. Due to dynamic nature of their field of study, the computer teachers need to continuously 

update their content knowledge. According to them, the main barrier was the insufficient in-service training 

opportunities. Therefore, computer teachers should be provided with rich and continuous in-service training. The 

teachers also expressed that in addition to the lack of well-equipped computer labs, insufficiency of pre-service 

teacher training programs, out-of-date computer curricula and large class sizes were increased the students’ 

indifferences and inattentiveness. These problems were also affected negatively the teachers’ effectiveness in the 

computer classes. Thus, pre-service computer teacher training programs should be reformed. The elementary and 

secondary schools’ computer curricula should frequently be revised. 

One computer teacher noted the following about his/her experience of “inadequate supervision and inspection”: 

Last year a supervisor visited one of my classes and just because one of the computers was not loaded 

with the MS Word (the students have removed it) he graded me the lowest. But, the fact is that now 

three of the computers are out of order in the computer lab and the school administration refuses to 

have them repaired claiming they don't have money to spare for it. Now I ask you, who is going to be 

responsible in this case? Is it my fault? (Message 286) 

According to the contemporary supervision approaches, supervisors should be a guide for teachers. However, 

supervisors’ lack of technological knowledge and skills may hinder their ability to make changes in their supervision 

practices. A possible solution could be to train supervisors both about the computer technology and the proper 

criteria for evaluating effectiveness of computer education. 

Social Support Analysis 

The content of computer teachers’ online discussion forum messages were also analyzed in terms of the types of 

social support provided. These categories are reported in Table 3. Majority of the messages (87.9%) were identified 

as providing with “emotional” support. Studies revealed that emotional concerns and support are widespread in 

CMC. In a study by Anderson and Lee (1995), it was found that beginning teachers offered emotional and moral 

support (personal concerns), rather than curricular and instructional advice (technical concerns) in their e-mail 

messages. Similarly, Nicholson and Bond (2003) found that pre-service teachers used the electronic discussion board 

by expressing many thoughts, experiences, and emotions. Over time it became a place of professional support and 

community. 

Table 3. The types of social support provided by messages 

Social Support Categories f % 

Emotional Support 

Feedback 410 75.5 

Approval/Praise 51 9.4 

Humor 16 3.0 

Subtotal 477 87.9 

Instrumental Support 

Informational 17 3.1 

No Social Support 49 9.0 

Total 543 100 

In the present study, the most frequent subtype of emotional support was feedback (75.5%) (e.g., teachers brought 

relevant personal experiences or thoughts into the discussion as a response to another message). Sharing personal 

experiences and thoughts has been considered to be useful for establishing and maintaining “group cohesion,” a key 

137

feature of supportive online groups. Babinski, Jones, and DeWert (2001) exposed that teachers were prone to express 

their immediate personal experiences and ask for advice in online discussion forums. The importance of knowing 

peers’ rich resources of practical knowledge in the professional development process is often emphasized. When 

professionals search for similarities from across the profession, it can “yield a fresh exchange of ideas, practices, and 

solutions to common problems” (Cervero, 1988, p.15 as cited in Anderson & Kanuka, 1997). Besides, Beehr and 

Glazer (2001) suggest that by conversing with others and learning about their experiences, people might learn to 

cope with their own stress factors. Therefore, the examined online discussion forum can be described as a very rich 

professional platform that teachers heavily exchanged their personal professional experiences and thoughts in order 

to draw attention to and find solution alternatives for their job-related problems. 

Other subtypes of emotional support were approval / praise (9.4%) (e.g., teachers expressed their emotional 

appreciation or praise as a response to another message), and humor (3%) (e.g., teachers told a joke or drew attention 

to an irony related to the topic being discussed). On the other hand, merely 3.1% of the messages were identified as 

providing “instrumental” (informational) support (e.g., teachers provided with specific information to their 

colleagues when they asked for) while 9% of the messages did not include any type of social support. 

In an ironic response to another teacher’s message who expressed his/her resentment against lowering the total 

teaching time of computer courses in the elementary schools, one computer teacher wrote: 

…I think one hour for computer courses is enough... Yes, that's right... You have heard it all right. 

What I am simply saying is that it takes at least a half hour to start the computer in our lab and another 

half hour to shut it down... (Message 406) 

Humor has been considered a good coping strategy (Austin, Shah, & Muncer, 2005). Coping however, refers to the 

stressed person’s own behavior, actions or intentions (Beehr & Glazer, 2001). When humor comes from other 

people, it can function as an emotional support that might make stressed people feel emotionally better. Furthermore, 

humor is one of the verbal immediacy behaviors that can lessen the psychological distance among users in an online 

discussion forum (Swan, 2002). The computer teachers used humor occasionally, yet effectively, to make their 

colleagues feel emotionally better and to air their problems. 

Because the computer teachers’ job-related problems mostly resulted from educational policies and organizational 

factors rather than the teachers’ personal deficiencies, both individual information seeking and informational support 

were extremely rare. However, in a case of a teacher asking for a specific type of information, his/her colleagues 

immediately provided him/her with such information. We observed that the teachers mainly asked for information 

from their veteran peers about their rights and responsibilities in schools, and potential solution alternatives for their 

technological problems in computer labs. These kinds of information exchanges among teachers are the instances of 

informational support. Here is an example of a reciprocal correspondence concerning this kind of information 

exchange: 

...The students delete program files of computers in order to prevent class work time and to instead 

engage in games. Could anybody help me to put an end to this? (Message 229) 

...Currently I use “Deep Freeze.” I highly recommend it. Also I think the “Ghost”, “NetOp School” or 

“NetSupport” could be useful... Here’s how you can set up and use the above mentioned software... 

(Message 248) 

Lastly, one computer teacher wrote the following to express his/her approval/praise about the content of ongoing 

discussions: 

...Thanks ever so much for the information you've passed on. I rather think deleting the program files 

by the students is as important as the other problems cited in the forum. To discuss in details and to 

explain which software could be used in order to put an end to such problem has certainly been useful 

to at least solve one of the problems we face. (Message 237) 

To provide a person with approval or praise concerning a particular behavior might improve his/her emotions such as 

self-confidence, self-esteem, and enthusiasm increasing the probability of performing this behavior. Approval and 

138

praise are also described as useful strategies to foster social interaction among participants in online settings (see 

Hara et al., 2000). By providing with such online emotional support, the teachers created positive communication 

and supportive relations among their colleagues, and helped them to feel emotionally better. 

Discussion and Conclusion 

This study examined the content of messages posted by Turkish computer teachers in an online discussion forum 

about their job-related problems. Findings suggest that computer teachers face a number of problems mostly resulted 

from educational policies and organizational factors such as role conflict, inadequate teacher induction policies, and 

lack of technological infrastructure and technical support. The frequency and variety of these problems indicate a 

need of a national comprehensive technology planning. According to Anderson (1999), an important step in 

technology planning is to face the honest reality of a condition, then to work together to build a strategy for success. 

Therefore, determining computer teachers’ job-related problems could provide the necessary data for the first step of 

technology planning. In this context, CMC can be an excellent channel for information exchange. Indeed, the value 

of CMC lies in its ability to facilitate professional collaboration between teachers and to encourage critical reflection 

on educational policy and practice (Hawkes & Romiszowski, 2001). Moreover, because of the anonymity, teachers 

can be more open and honest in CMC environments, especially in online forums. Therefore, educational policy and 

decision makers might benefit from CMC as a feedback mechanism to analyze the context, determine the needs and 

specify the goals for successful implementations. System developers must consider designing an online system that 

provides rapid information flow from school teachers to the central and/or local policy centers. 

Another noteworthy finding of this study is that the computer teachers mostly share their personal-professional 

experiences and thoughts via online discussions. These personal experiences and thoughts and emotional support 

were very common in the reciprocal online discussions. Therefore it can be concluded that the online social 

interaction among the computer teachers was quite reflective. Reflectivity is considered to be the heart of 

professional development. Reflection is a continual process that engages teachers in framing and re-framing 

problems while designing and evaluating solutions (Hawkes & Romiszowski, 2001). Reflective teachers tend to 

examine and re-examine their personal-professional experiences to improve their teaching practices and working 

conditions. The present study confirmed that the Turkish computer teachers used the online discussion forum as a 

social-professional platform for sharing their job-related problems, suggesting potential solutions, and providing 

and/or receiving social support. Professional development for teachers constitutes formal and informal processes of 

knowledge and skill building (Hawkes & Romiszowski, 2001). In this context, CMC tools have the potential to 

become rich, flexible, formal or informal, and personal learning environments (Attwell, 2006; Downes, 2006). For 

example, case libraries, online libraries, video-cases, online technical services, special interest groups, access to 

written regulations, social support groups, and open-access curriculum materials are such applications that can be 

available for teachers in CMC environments. 

Finally, content analysis technique provides an invaluable tool to understand the nature of communication and social 

interaction patterns among users in online environments. It is hoped that the findings of this study could be helpful in 

stimulating educational researchers to pay attention to alternative data sources such as communication transcripts of 

teachers’ discussion forum messages to better understand what types of problems they have and what types of 

support they need. Further studies are needed to uncover several points including what are the essential role of 

computer teachers, and to what extent they felt supported or became relieved as a result of online discussions. Lastly, 

the variables that can affect the functionality and viability of online groups such as group size, group compositions, 

and participation degree of users are the issues needing further exploration. 

References 

Abel, M. H., & Sewell, J. (1999). Stress and burnout in rural and urban secondary school teachers. The Journal of 

Educational Research, 92 (5), 287-293. 

Anderson, L. S. (1999). Technology planning: It’s more than computers, retrieved on October 15, 2007, from 

http://www.nctp.com/articles/tpmore.pdf. 

139

Anderson, J., & Lee, A. (1995). Literacy teachers learning a new literacy: A study of the use of e-mail in a reading 

instruction class. Reading Research and Instruction, 34, 222-238. 

Anderson, T., & Kanuka, H. (1997). On-line forums: New platforms for professional development and group 

collaboration, ERIC Document Reproduction Service No. ED- 418693. 

Attwell, G. (2006). Personal learning environments, retrieved on October 15, 2007, from 

http://www.knownet.com/writing/weblogs/Graham_Attwell/entries/6521819364. 

Austin, V., Shah, S., & Muncer, S. (2005). Teacher stress and coping strategies used to reduce stress. Occupational 

Therapy International, 12 (2), 63-80. 

Babinski, L. M., Jones, B. D., & DeWert, M. H. (2001). The roles of facilitators and peers in an online support 

community for first-year teachers. Journal of Educational and Psychological Consultation, 12 (2), 151-169. 

Beehr, T. A., & Glazer, S. (2001). A cultural perspective of social support in relation to occupational stress. In P. 

Perrewé & D. C. Ganster (Eds.), Research in occupational stress and well-being, Greenwich, CO: JAI Press, 97-142. 

Berelson, B. (1952). Content analysis in communication research, Glencoe, IL: Free Press. 

Black, S. (2003). Stressed out in the classroom. American School Board Journal, 190 (10), 36-38. 

Brissie, J. S., Hoover-Dempsey, K. V., & Bassler, O. C. (1988). Individual, situational contributors to teacher 

burnout. Journal of Educational Research, 82 (2), 106-112. 

Brown, S., & Nagel, L. (2004). Preparing future teachers to respond to stress: Sources and solutions. Action in 

Teacher Education, 26 (1), 34-42. 

Burke, R. J., & Greenglass, E. R. (1993). Work stress, role conflict, social support and psychological burnout among 

teachers. Psychological Reports, 73, 371-380. 

Burleson, B. R., & Goldsmith, D. J. (1998). How the comforting process works: Alleviating emotional distress 

through conversationally induced reappraisal. In P. A. Anderson, & L. K. Guerrero, (Eds.), Handbook of 

communication and emotion: Theory, research, application, and contexts, San Diego, CA: Academic Press, 245-280. 

Caplan, S. E., & Turner, J. S. (2007). Bringing theory to research on computer-mediated comforting communication. 

Computers in Human Behavior, 23 (2), 985-998. 

Cervero, R. (1988). Effective continuing education for professionals, San Francisco, CA: Jossey-Bass. 

Cohen, S., & Wills, T. A. (1985). Stress, social support, and the buffering hypothesis. Psychological Bulletin, 98, 

310-357. 

Cooper, C. L., & Marshall, J. (1978). Understanding executive stress, London: Macmillan. 

Deryakulu, D. (2005). Bilgisayar öğretmenlerinin tükenmişlik düzeylerinin incelenmesi. Eğitim Araştırmaları, 19, 

35-53. 

Deryakulu, D. (2006). Burnout in Turkish computer teachers: Problems and predictors. International Journal of 

Educational Reform, 15 (3), 370-385. 

De Wever, B., Schellens, T., Valcke, M., & Van Keer, H. (2006). Content analysis schemes to analyze transcripts of 

online asynchronous discussion groups: A review. Computers & Education, 46, 6-28. 

Downes, S. (2006). Learning networks and connective knowledge, retrieved October 15, 2007, from 

http://it.coe.uga.edu/itforum/paper92/paper92.html. 

140

Eastin, M. S., & LaRose, R. (2005). Alt.support: Modeling social support online. Computers in Human Behavior, 21, 

977-992. 

Evers, W. J. G., Brouwers, A., & Tomic, W. (2002). Burnout and self-efficacy: A study on teachers’ beliefs when 

implementing an innovative educational system in the Netherlands. British Journal of Educational Psychology, 72, 

227-243. 

Farber, B. A. (1984). Stress and burnout in suburban teachers. The Journal of Educational Research, 77, 325-331. 

Friedman, I. A. (1991). High-and-low-burnout schools: School culture aspects of teacher burnout. Journal of 


Guglielmi, R. S., & Tatrow, K. (1998). Occupational stress, burnout, and health in teachers: A methodological and 

theoretical analysis. Review of Educational Research, 68, 61-99. 

Hara, N., Bonk, C. J., & Angeli, C. (2000). Content analysis of online discussion in an applied educational 

psychology course. Instructional Science, 28, 115-152. 

Hawkes, M., & Romiszowski, A. (2001). Examining the reflective outcomes of asynchronous computer-mediated 

communication on inservice teacher development. Journal of Technology and Teacher Education, 9 (2), 285-308. 

Hendley, D., Stables, S., & Stables, A. (1996). Pupils’ subject preferences at key stage 3 in South Wales. 

Educational Studies, 22, 177-186. 

Kyriacou, C. (2001). Teacher stress: Directions for future research. Educational Review, 53 (1), 27-35. 

MEB. (2004). Temel eğitim ikinci faz başlangıç semineri, Ankara: Imaj. 

MEB. (2005a). Bilgisayarlı eğitime destek kampanyası başladı, Retrieved on October 15, 2007, from 

http://www.meb.gov.tr/. 

MEB. (2005b). Milli eğitim istatistikleri 2004-2005, Ankara: Devlet Kitapları Basımevi. 

Naidu, S., & Järvelä, S. (2006). Analyzing CMC content for what? Computers & Education, 46, 96-103. 

Nicholson, S, A., & Bond, N. (2003). Collaborative reflection and professional community building: An analysis of 

preservice teachers’ use of an electronic discussion board. Journal of Technology and Teacher Education, 11 (2), 

259-279. 

Olkun, S., Altun, A., & Smith, G. (2005). Computers and 2D geometric learning of Turkish fourth and fifth graders. 

British Journal of Educational Technology, 36 (2), 317-326. 

Pines, A. M. (2002). Teacher burnout: A psychodynamic existential perspective. Teachers and Teaching: Theory and 

Practice, 8 (2), 121-140. 

Rourke, L., & Anderson, T. (2004). Validity in quantitative content analysis. Educational Technology Research & 

Development, 52 (1), 5-18. 

Rourke, L., Anderson, T., Garrison, D. R., & Archer, W. (2001). Methodological issues in the content analysis of 

computer conference transcripts. International Journal of Artificial Intelligence in Education, 12, 8-22. 

Russell, D. W., Altmaier, E., & Van Velzen, D. (1987). Job-related stress, social support, and burnout among 

classroom teachers. Journal of Applied Psychology, 72 (2), 269-274. 

Schwab, R. L., & Iwanicki, E. F. (1982). Perceived role conflict, role ambiguity, and teacher burnout. Educational 

Administration Quarterly, 18, 60-74. 

141

Schwab, R. L., Jackson, S. E., & Schuler, R. S. (1986). Educator burnout: Sources and consequences. Educational 

Research Quarterly, 10 (3), 14-30. 

Swan, K. (2002). Building learning communities in online courses: The importance of interaction. Education, 

Communication and Information, 2, 23-49. 

van Dick, R., & Wagner, U. (2001). Stress and strain in teaching: A structural equation approach. British Journal of 

Educational Psychology, 71, 243-259. 

Walther, J. B., & Parks, M. R. (2002). Cues filtered out, cues filtered in: Computer-mediated communication and 

relationships. In M. L. Knapp, J. A. Daly, & G. R. Miller (Eds.), The handbook of interpersonal communication (3 rd 

Ed.), Thousand Oaks, CA: Sage, 529-563. 

Weisskirch, R. S., & Milburn S. S. (2003). Virtual discussion: Understanding college students’ electronic bulletin 

board use. Internet and Higher Education, 6, 215-225. 

Wolpin, J., Burke, R. J., & Greenglass, E. R. (1991). Is job satisfaction an antecedent or a consequence of 

psychological burnout? Human Relations, 44 (2), 193-209. 

Wright, K. (2000). Computer-mediated social support, older adults and coping. Journal of Communication, 50, 100- 

119. 

142

Johnson, R., Kemp, E., Kemp, R., & Blakey, P. (2007). A theory for eLearning. Educational Technology & Society, 10 (4), 143- 

155. 

The learning computer: low bandwidth tool that bridges digital divide 

Russell Johnson, Elizabeth Kemp, Ray Kemp and Peter Blakey 

Massey University, Palmerston North, New Zealand // R.S.Johnson@massey.ac.nz // E.Kemp@massey.ac.nz // 

R.Kemp@massey.ac.nz // P.Blakey@massey.ac.nz 

ABSTRACT 

This article reports on a project that explores strategies for narrowing the digital divide by providing a 

practicable e-learning option for the millions living outside the ambit of high performance computing and 

communication technology. The concept is introduced of a learning computer, a low bandwidth tool that 

provides a simplified, specialised e-learning environment which works with or without an internet connection. 

This concept is contrasted with the Learning Management System model widely adopted by universities, in 

which teaching material is accessed as web pages from a central repository. The development of an initial 

prototype and its field testing under realistic conditions is reviewed, and plans for future work outlined. 

Keywords 

Distance learning, Digital divide, Low bandwidth, Accessibility, Ubiquitous computing 

Motivation for the project 

Over the past decade, mechanisation has had a major impact upon distance learning, with computer-based courses 

delivered over the World-wide Web increasingly replacing traditional correspondence courses. As rapid advances in 

computer and communication technology made possible real-time video-conferencing and virtual classrooms via the 

PC many began to question the future of “bricks-and-mortar” learning institutions, promoting e-learning as the future 

of education in an internet-linked global village. 

Despite this potential, it is widely acknowledged that the boom in computing and communication resources is 

actually deepening the digital divide between people who have access to and know how to use technology and those 

who do not (Bork, 2001; Bindé, 2005; Marine & Blanchard, 2004). This divide exists between developed and 

developing countries, but also between urban and rural regions within every country. As recently as 2002, one 

communication studies researcher observed, online information services such as e-learning were still focussed upon a 

few hundred “wired cities”, while ninety-seven percent of the world’s population had no Internet access and almost 

two-thirds of households had no telephone, let alone the broadband delivery and multimedia computer required for 

virtual classrooms (Hope, 2002). 

The learning computer project began as a search for ways in which information technology could be applied to 

distance learning so as to reach beyond the “wired cities” to narrow this digital divide in education and to enhance 

all distance students’ learning experiences. In harmony with distance education’s goal of providing learning 

opportunities to those unable to attend conventional learning institutions by reason of geographic location, job, 

disability or age, we were seeking a path towards anybody, anywhere, anytime e-learning. Special factors that had to 

be considered included poor internet service, older computers, and isolated users struggling to complete computing 

tasks unaided. 

Anecdotal evidence and personal experience suggested that the commercially-available web-based Learning 

Management Systems (LMS) were too reliant on higher end technologies, too complex for a relatively novice user to 

negotiate unaided, and too pedagogically-limited to meet these requirements. Using LMS’s to replace 

correspondence courses may actually perpetuate and even widen educational inequalities through creating a two-tier 

system based on access to, and ability to use, the technology. 

This article reports on the results from the first three stages of our project. First we began by investigating the 

problem. From a review of prior research we identified some key criteria for designing an effective computersupported 

distance learning system. Second we conceptualised an e-learning system that met these criteria. Then we 

tested this conceptualisation through an initial prototype which we evaluated with users. After reporting on these 

stages the article draws some initial conclusions and identifies the further steps in evaluating our approach. 






143

Investigating the problem 

An extensive review of conference papers, journal articles, books, web sites, industry publications, news reports and 

software was conducted, highlighting the rich research record in this field. The main results of our review have been 

published previously (Johnson et al, 2002). Here we can only briefly summarise those issues most relevant to our 

conclusions. 

“Distance learning” covers a broad range of scenarios with very different requirements – from specialised in-house 

job training, through radio schools for children in isolated rural areas, to open university adult education courses. For 

this project, it was necessary to focus upon a specific sector: the computer-based delivery of university-level, home 

study programmes for students unable to attend normal lectures. 

Computer systems, through their ability to rapidly store, manipulate and communicate multimedia information, have 

the potential to provide multi-dimensional distance learning environments, which incorporate passive (teachercentred) 

or active (student-centred) learning styles, group or individual work, interaction, and simulation. However, 

such environments have not been fully realised at the university-level. Moreover, e-learning systems often assume 

levels of computer literacy and access to reliable, high-speed technology more in line with the resources of public or 

private institutions than with students studying at home or in developing countries. 

All the systems in wider use at the university level were built upon the LMS model. This is a “one-size-fits-all” 

approach in which teaching material is stored on a central repository and delivered page by page over the Internet to 

a web browser on the learner’s computer. The learner must establish a live connection to the university server, and 

then wait until the next page is downloaded before he/she can proceed to study. 

The LMS’s originated as practical tools to aid university staff to author and administer their internal and external 

courses e.g. WebCT (Goldberg, 1997) and Blackboard (Kubarek, 1999). A major distinction is now drawn between 

LMS’s and Learning Content Management Systems (LCMS). The Brandon Hall website, for example, explains that: 

"The primary objective of a learning management system is to manage learners, keeping track of their progress and 

performance across all types of training activities. By contrast, a learning content management system manages 

content or learning objects that are served up to the right learner at the right time" (Brandon Hall, 2005, para. 2). 

From this perspective, the primary target users of an LMS are "training managers, instructors, administrators", while 

for an LCMS they are "content developers, instructional designers, project managers" (ibid, para. 5). However, from 

the standpoint of the usability and accessibility of the system to distance students there is no essential difference 

between these models. To access learning material, the student must correctly navigate the operating system, the web 

browser and the multi-page web application. None of these tasks is trivial. Needing to learn them can inhibit learning 

the course content (Smulders, 2003). 

Some LMS’s have been prototyped that address the accessibility issue by acting as a standalone system when an 

internet connection is unavailable, including the TILE system developed at Massey University (Jesshope et al., 

2000). This hybrid model does not address the usability issues, however. 

There are some good examples of one-off adaptive tutoring systems at the university-level. While offering improved 

functionality over a LMS, they have proven too complex to implement to come into general use. More recent 

research efforts have focussed upon incorporating adaptive elements into LMS’s and LCMS’s (e.g. Jong et al., 

2003). 

It was also noted that e-learning has been most effective when integrated with on-campus teaching or research. 

Attempts to replace teacher and textbook altogether and create entirely electronic virtual universities have largely 

failed (Education Review, 2004). 

Mobile e-learning, or “m-learning”, is a growing research area. M-learning explores ways of using mobile devices 

like PDA’s and cellular phones for supporting and/or delivering some elements of teaching and learning processes, 

especially with a view to drawing young people into learning (Attewell, 2004). While a cellular phone is a powerful 

networked computing device its very short transmission range and restricted interface limit its potential for extending 

the boundaries of e-learning into rural areas. 

144

From our search for technologies that could be generalised and implemented as a practical alternative to the LMS 

model, we identified eight key criteria that our system would have to meet. These were: 

1. Target the distance learner as primary user. Systems with a focus on course authoring or administration, 

supporting on-campus study, or providing in-house job training, have different requirements from those oriented 

towards the distance university student. 

2. Prioritise the student view. Designing as a learning environment means focussing upon the content, interactions, 

presentation styles and user support at the student interface. A learning environment integrates all the functions 

and features that support the student's learning tasks. It treats the student view as the front end of a learning 

system rather than as the back end of a teaching application. 

3. Recognise the university itself as the most important learning community. The learning environment extends the 

university’s reach to the distance student. This means facilitating a range of learning features and modes, which 

complement rather than replace the roles of the teacher and the textbook, encourage classmates to work together, 

and draw students into the university learning community. 

4. Provide for the environment to adapt or be adapted to the individual learning task or student. This may include 

how and what learning material is presented when, help in the use of one or other learning features, querying of 

subject matter, and assistance in locating supporting material. However, it may not be desirable or feasible to 

individualise all learning features. And sometimes the individual learner should adapt to facilitate collaborative 

work and discussion between students. 

5. Design as an information system. The learning environment is best conceived of as an information system 

incorporating hardware, software and human factors, in which automation is not always assumed to be better. A 

decision is made on what to computerise and what not to, based upon what works best for the distance learner. 

Some aspects of a course may be better implemented using alternative technologies to the PC or a human 

teacher. 

6. Design for reusability on three levels - the programmer, the course author, and the student. For an extramural elearning 

system to provide a practical alternative, it must not only be good for student learning, it must also 

efficiently support authoring, managing and delivering multiple courses of study. In addition, it should be able to 

readily integrate new and improved learning technologies. 

7. Consider alternative delivery technologies. The Internet, as the backbone of any networked e-learning system, 

embraces synchronous and asynchronous alternatives to the ubiquitous web-server/browser-client model. A 

specialised interface may be more effective for learning than a general purpose web browser. One-way satellite 

broadcasting has a much wider geographical reach than cellular or landline telecommunications. 

8. Evaluate for functionality, usability and accessibility. In evaluating the system's reliability and performance a 

lowest common denominator approach is required. All the key functions and features should not only work with 

top of the range computers and high-speed urban communication networks, but should be tested for usability 

and accessibility in more challenging circumstances like that of isolated rural students. Interactive and 

multimedia features delivered over the Internet may not be available to the remote student in a timely manner. 

Conceptualisation and specification of a learning computer 

The most fundamental conclusion we drew from our review was that for an e-learning system to successfully support 

university distance education the focus had to shift from the education-provider to the distance learner as the primary 

user. In particular, some research results suggested to us that our objectives of maximising the learning functionality 

available to the distance student, while minimising the usability and accessibility problems of web-browser-based 

systems, could be better met through user-centred strategies of specialisation, customisation, and localisation. By this 

we mean: 

• That special-purpose computer tools (interfaces, programming languages, hardware...) offer a way of providing 

a simpler and more usable learning environment than general-purpose ones (web browser, HTML, PC…) by 

helping to render the computer invisible to the learning process (Bork, 2001; Hoppe et al., 2000); 

• That user-oriented interface design, incorporating customisable and collaborative strategies, can more easily and 

simply achieve many of the individualisation and interactivity goals of adaptive systems, while maintaining the 

locus of control with the student (Murray et al., 2000); and 

• That by placing more of the system's functionality on the student’s computer and then using the internet or 

alternative media to update the student’s computer, the greater functionality associated with standalone systems 

and the collaborative benefits of networked computers can be incorporated into an e-learning system that better 

145

meets the anywhere, anytime requirements for distance study than server-centred systems (Dietinger & Maurer, 

1998; Jesshope et al., 2000). 

By considering these strategies together, the concept crystallised of a learning computer – a special-purpose 

computational device which integrates all the features a student needs for learning into a simplified, specialised, 

individualised, networked machine. The distinctive elements in the learning computer model include: 

• Learner-centred – all system features are designed from the student end back, and retain the locus of control 

with the student. 

• Asynchronous – no essential communications (student-teacher, student-student, system-system, etc.) require 

completion in real-time. 

• Integrated – seamless conflating of operating system, browser and application functionality through a specialised 

interface. 

• Decentralised – functionality and content are distributed to the learner end, requiring only periodic updating. 

• Modular – system functionality and content are built up from interchangeable modules to support course 

customisation and reuse. 

The learning computer provides alternative study modes, defined by a set of elements, each of which corresponds to 

a basic constituent of university study. Learning by lecture, for example, may involve a presentation by the lecturer, 

a set of slides provided by the lecturer, and notes made by the individual student. Learning by tutorial, on the other 

hand, may involve a one-to-one interaction with a tutor as well as notes made by the individual student. 

Having visualised the new system in this way, it was then necessary to specify a prototype in more detail. The key 

decisions made at this stage were: 

• That the learning computer would be implemented as the client in a client/server network in which most 

functionality is distributed to the client side. Communications and learning content would be updated by 

periodically linking to a central repository of messages and learning resources via the Internet or one-way 

satellite download (IHug, 2004) using the FTP protocol, or via the post (i.e. CDs), rather than as a browser client 

downloading pages from a web server. Updates could then proceed in the background while the learning 

computer is in use. 

• That the student interface would be implemented on the Windows platform as a specialised graphical Learning 

Shell replacing the operating system GUI, rather than by adding functionality to a general-purpose web browser. 

This Shell would provide "just-enough" of the underlying Windows functionality "just-in-time" to support the 

required learning task. It would be assembled on a modular basis using an object-oriented rapid application 

development environment that supported reusable software components. Delphi was chosen for this. 

• That navigation and other user interactions with the Shell would be simplified by managing them through an 

intermediate system layer between the interface and the operating system, which models the modular, 

hierarchical structure of a distance learning course and maps the user’s learning tasks to the appropriate 

computer operations. This layer would also model the student sufficiently to provide the reference point for 

individualising the Shell to a specific user. 

• That learning assistance would be provided to the student through an integrated system of electronic mail, 

collaborative work groups and online help built around a replicated SQL-compliant database. The online help 

system would be queried by the student via a user-friendly interface to obtain more information on an issue or 

guidance on where to find more information. In this way the learner obtains assistance through discussion with 

other students, contacting the course tutor, or by directly querying the database. Student queries and their 

outcomes could be logged to facilitate dynamically improving the support system. 

To implement these decisions and evaluate the overall concept, a course authoring and management application and 

a communications management component needed to be built and integrated with the learning computer into a 

network. This networked prototype we have called IMMEDIATE (Integrating MultiMEdia in a DIstAnce learning 

and TEaching environment). 

146

Evaluation through prototyping and user testing 

Incremental prototyping was used to evaluate and refine the main elements of the design specification and to develop 

IMMEDIATE into a fully operational network. The approach taken was to focus upon meeting the requirements of 

the Learning Shell as the front-end of the system and then to address networking, authoring and administration issues 

in the context of supporting those requirements. 

The Learning Shell user interface has been assembled in Delphi from reusable “drag and drop” software components. 

Each of these encapsulates a particular learning element such as a student notebook or a lecture presentation. They 

are built to a template to ensure consistency across the system and are self-contained in terms of learning 

functionality, only needing to call the Shell API to access or update learning content. An algorithm maps each 

learning content file to a particular learning component and topic in the course, enabling the system to synchronise 

components by topic and provide context-sensitive help. The learning support system is built upon a database of 

definitions, elaborations, references and URLs linked to keywords associated with each topic in a course. This 

components-based architecture provides a robust, modular structure capable of incorporating more advanced 

technologies as appropriate. 

As a network user interface in an internet-based client/server network, the Learning Shell has some functional and 

architectural similarities to a platform-specific web browser such as Microsoft's Internet Explorer that is tightly 

linked to the underlying operating system. It is built around a Controller module providing an API through which 

interface components interact with each other and with the reference model layer (Figure 1). The model layer is a set 

of data structures and related operations, each of which models a different aspect of the learning computer and its 

current state. The System Model, for example, defines the different study modes available in a course, and maps 

individual learning components to each mode. 

Desktop 

Enables user to 

interact with the 

system controller 

to change modes, 

launch interface 

components, etc. 

Operating 

System API 

Called by 

components 

for their 

internal 

functionality 

User Info 

Enables system to 

directly collect info 

from the user - logon, 

backup drive,etc. 

Generic Learning 

Component 

Tools available in any 

mode. Managed by 

the user, e.g. Student 

Notes 

Manager Manager Manager Manager 

Student Model 

Holds student 

info needed by 

system & not 

stored 

elsewhere, 

e.g. current 

topic & mode, 

password 

System Model 

Maps 

components to 

modes and 

holds other 

static info 

about system 

System Tree 

Maps structure 

of course. 

Stores 

dynamic info - 

nodes (topics) 

visited or 

completed, etc 

Figure 1. Learning Shell architecture 

Resources 

Model 

Maps stored 

files to 

learning 

components 

Mode-Specific 

Learning Component 

Accessible only by 

changing mode - 

managed by system 

e.g. Lecture Notes. 

Controller 

All interactions between components are channelled through here. Manages the 

interface components - mode changes, etc. Interacts with system components via 

respective managers. 

Learning shell directory system 

Manager 

System 

Database 

Integrates 

messages & 

learning help 

The Controller plays an analogous role to a browser’s controller, which manages the other browser components and 

calls on them to perform operations specified by the user (Comer, 1999, p. 427). However, the Learning Shell 

Controller, through reference to the model layer, has additional capacities for managing the interface to simplify and 

147

minimise housekeeping tasks and free the user for learning. In tandem with the learning components architecture it 

provides a direct manipulation interface which seamlessly integrates all necessary functionality into a single, 

simplified, adaptable framework. In contrast to a cross-platform browser environment learning components can call 

the Windows API directly, taking full advantage of its rich interactive functionality. 

Network architecture 

The IMMEDIATE network architecture is shown in Figure 2. The university end of IMMEDIATE uses an FTP 

server as its gateway, controlling access to the Course Repository. Because most functionality resides at the student 

end the only materials that need to be transferred across the network are messages and updated learning or system 

management files. Most updates are short text files (messages and help updates) which can be readily transferred 

even over slow Internet connections. Where land-based Internet service is too slow or unreliable, larger files can be 

updated by alternative media such as CD or DVD disks or satellite. FTP offers a simple basis for meeting these 

requirements and enables updates to proceed behind the scenes without the user having to wait for downloads to 

complete. 

To support a communications model based on a replicated database, each user's view of the database is copied to that 

user's machine from a central master copy. A set of protocols has been devised and implemented for updating and 

synchronising these views and the master copy over FTP. All updates are SQL result sets which are transferred as 

plain text files. 

The course authoring, tutoring and administration application was built to prototype a method for assembling and 

managing courses that does not require the tutor to understand IMMEDIATE's inner workings, and to help evaluate 

the communication and learning support capabilities of the learning computer. It uses the same mapping algorithms 

and data structures as the Learning Shell to enable a course to be outlined, study modes to be defined and learning 

content to be inserted correctly via a direct-manipulation, graphical interface. This mechanism enables course 

content to be dynamically improved, updated, and re-used after it has been deployed. 

Course Management & 

Authoring System 

(Tutor User) 

On-campus 

Off-c ampus 

LAN 

Reposit ory 

(Database, 

update files) 

Repository 

Manager 

FTP Server 

(Security) 

Internet 

Learning Shell 

(St udent us er) 

Figure 2. IMMEDIATE network architecture 

CD-ROM 

148

The final incremental prototyping step was to install IMMEDIATE’s component applications on a LAN to ensure 

that they would work correctly together to support the key learning computer requirements. Once we had achieved 

this our next task was to evaluate these features under field conditions. 

Evaluation with users 

IMMEDIATE was installed to run over the rural telephone network and then tested with volunteer users from a 

remote farming and fishing community over a period of two days. This involved setting up the Learning Shell on a 

PC, the selection of four volunteers, a pilot study with one user and an in-depth study with the other three. If 

IMMEDIATE was to have universal application then it was vital to test it under some of the more difficult field 

conditions that might be faced by distance students. It was necessary to ensure that the essential functionality of the 

system was accessible with an older model PC and operating system, a slow and unreliable rural Internet connection, 

and a relatively inexperienced computer user working alone. 

The goals of this evaluation were to determine that IMMEDIATE performed reliably under these conditions and that 

all of the learning computer's functionality was accessible to the user, and to assess the usability of the Learning 

Shell. Developers of any computer-based learning must consider students both as learners and as users, that is, they 

must design both for form and content (Smulders, 2003). Poor usability inhibits learning. Inaccessibility prevents it 

altogether. This is a critical issue where the student is studying at home alone without access to user support. 

In this phase of the project, the evaluation was focussed on the student as user, that is, on the technical question of 

whether the users could carry out the main tasks, without considering the pedagogical effectiveness of the learning 

computer. This involved evaluating the functionality of all three subsystems – the Repository Manager, the Course 

Authoring and Management System and the Learning Shell. For students to carry out the specified tasks, all the three 

subsystems had to work together effectively. 

The usability of the student end also had to be evaluated. Could the participants complete a set of typical learning 

tasks unaided? How efficiently could they do this? 

Finally, the following questions needed to be addressed with regard to accessibility. Was the system runnable where 

the Internet was slow and unreliable? Could key functions, such as updating resources, communicating with other 

students and getting help from the tutor, work over a rural network? 

The Learning Shell was installed on PC running Windows 95 at a small rural school. Four members of the 

community serviced by the school volunteered to assist in evaluating the usability of the prototype. All four 

volunteers had studied at the tertiary level and three were current or very recent tertiary distance students. One 

agreed to make herself available for a pilot study which was used to refine the evaluation to ensure that all major 

aspects of the system would be tested. 

The three remaining volunteers participated in the in-depth study. They were selected on the basis that they lived in a 

rural area, had distance learning experience and some knowledge of computers (the relevant information was 

obtained by a questionnaire). This is an example of purposive sampling (Yin, 1984, Patton, 1990) where appropriate 

individuals who meet the specified requirements are selected. It was necessary to find individuals who had the 

potential to successfully complete the exercises under conditions similar to those faced by a distance student, 

working unassisted from a remote location. 

The three volunteers were asked to complete seven scenarios covering setting up the student Learning Shell and 

exploring all aspects of its functionality, including accessing learning material in six different study modes: lecture, 

group work, tutorial, textbook, collaboration, and practice. They were expected to complete these in two one-hour 

sessions spread over two days. The first exercises were quite detailed in their instructions, the later ones, 

progressively less so. To ensure that no user was advantaged by knowledge of the course content, the material was 

taken from a third year university paper on a topic that none of them had studied. This was appropriate given the 

nature of the evaluation. The sequence of scenarios was as follows: 

1. Initialise Learning Shell to own preferences 

2. Log onto course 

149

3. Join Group Discussion 

4. Attend Lectures, Seek Help, and Complete Self-Assessment 

5. Monitor the Assignment Discussion 

6. Explore On-line resources 

7. Complete Individual Tutorial 

One issue that had to be faced was of training. When the system was fully developed, student users would be 

supplied with a CD or DVD containing the software and course materials, a demonstration video and a hardcopy 

guide. Since at this stage there was no demonstration video, some training had to be provided to the users. Each user 

was given a demonstration of the student software as it would be shown in the video and then completed the first two 

exercises under supervision. They were then left to complete the scenarios unassisted, except for the provision of a 

sheet of Handy Hints that would form part of the hardcopy guide. 

Three forms of data collection were used during the experiment: logging their activities (Dix et al., 1993), 

observation and interviews (Patton, 1990, Scott et al, 1991). Where, when and what each user did during their 

sessions was tracked by the system and saved in a log file, primarily to analyse users' navigation paths and 

completion times, and record where they ran into difficulties. Because the student software is designed to support 

user-centred exploration and multiple navigation paths, a user should be able to complete a task even it they diverge 

from the shortest path one would expect an experienced user to follow. 

Each participant was interviewed individually, immediately after they had completed each session (i.e. twice). Semistructured 

interviews took place. A set of questions was prepared as the starting point for exploring the person's 

experience with the prototype. Questions such as the difficulties faced, what aspects of the system they liked, and 

what they would want changing were asked. These interviews were recorded and transcribed. 

Results of the Evaluation 

IMMEDIATE ran successfully and without major incident under quite challenging conditions. The results of this 

evaluation were very positive, with all volunteers able to complete the exercises within the two one-hour sessions. 

All three stated that by the end of the exercises they were confident that they were able to use the system unaided. 

The users were only observed working through the first two scenarios. The data from this observation was of 

somewhat secondary value because these two exercises were part of the supervised demonstration. However it did 

confirm the data from their profile questionnaires indicating that they represented a range of computing experience 

from confident to unsure. Two of the users were inhibited by previous negative experiences with Windows, 

including the fear of crashing the system if they did something wrong. 

An analysis of the log files revealed that whilst each of the volunteers got held up at least once they were able to 

complete all their scenarios. User 2, the least experienced user, took significantly longer to complete several 

scenarios (Figure 3). 

The most fruitful form of data collection proved to be the interviews which were undertaken with each user 

individually at the end of each one-hour session. Each volunteer was asked the same set of questions, as the basis for 

exploring their experience with the prototype. The interviews averaged between 20 and 30 minutes. No major 

usability problems were identified during the interviews. 

The complete interviews were transcribed and then analysed for significant, common themes relating to the goals of 

the evaluation. Major themes identified were: 

• The complexity of the general-purpose PC environment. 

• The problem of poor Internet service in rural areas. 

• The negative effects of personal isolation. 

• The importance of training and help. 

• Simplifying searches to avoid information overload. 

Support for the Learning Shell. 

150

Figure 3. Comparison of scenario completion times 

Many of the interviewees' comments alluded to the frustrations they faced using their computers at home. These 

comments strongly support the argument that modern personal computing systems have become too complex for 

learning (“there's a lot of stuff that you just don't need. And to get around to doing the task that you want to do, it's 

harder to get there because you've got obstacles in the way.”); that poor rural internet service renders server-centred 

systems unworkable (“waiting, waiting”; "You can dialup and get online. But you can't get anything."); and that 

without anyone more experienced on hand to help, the home student may not be able to complete their learning tasks 

(“It's easier to do the basic stuff, [easier to drive the 70 kilometres into town to] go to the library and things like that 

and photocopy pages out of a library book [than find the item on the web]. But it should be easier on a computer but 

I find that it is not.”). 

Overall the interviewees embraced the learning computer as an easier and more effective vehicle for achieving tasks 

related to e-learning. "Half the time I get frustrated when I have to work with my computer... If I was to be studying 

at Massey and there was a program like that, that would probably be one of the main reasons why I would study 

online… there were things that I got confused on but overall it was very clear what I was doing to the point where I 

thought: this is too clear, and too easy.” Some of the features they singled out were: simplicity, ease of use, 

integration, speed, timeliness, relevance, robustness, friendliness, transparency, consistency, and clear instructions. 

One user liked the way the Shell was simplified and standardised across all screens. Another noted that the 

advantages of this simplicity are that “you don’t have to remember a lot of things” and that “you don't have to worry 

about all the bells and whistles and everything else going on because it is all just there”. 

One of the benefits of the Learning Shell's simplified, consistent, modular architecture is that help screens, 

emphasising a "Handy Hints" approach, can be easily linked to each component in the system to provide just-in-time, 

just-enough, context-sensitive help to the user (Figure 4). For one user, “if I click on Help I have found every time 

what I need to do and its been able to solve it for me,” while accessing the help system in Windows applications 

didn’t solve his problem because “quite often it's not the scenario where you have got yourself”. Another noted that 

“the content is relevant to what you are doing" whereas on her home computer it was often “irrelevant drivel”. 

151

Figure 4. Learning Shell showing Help screen for Lecture component 

The volunteers also commented favourably on how the Learning Support system helped them rapidly locate 

supporting material compared to using a general web search engine where you "get 300 sites of which 275 of them 

are useless". "It was easy and it was fast. And that is my two main things at home… Because you didn't have to 

waste all that time in sorting through the information that you are given as to what's relevant and what's not." 

Comments were also solicited on how the system might be improved. One proposal was to add an “Undo" function 

by which the user could step back from their last action. 

Whilst there were only three users in the in-depth evaluation, they were representative of distance learners in remote 

environments. The number of users was sufficient to demonstrate that the concept was workable, that the Repository 

Manager could handle communication between the university and student end, that the Course Authoring and 

Management system could assist a remote student and that the system was operational over a two day period. The 

system proved accessible to users providing speedy access in contrast to the long delays that occur when using the 

Internet. Whilst the users took varying lengths of time to accomplish the tasks, all were able to complete them even 

though the course material was unfamiliar. In their comments the participants all acknowledged the benefits of a 

system of this kind. They were able to run this distributed system over a telephone network and demonstrate that the 

main functions could all be used. 

Conclusion and discussion 

The main result of this project so far has been to demonstrate the feasibility of the learning computer concept as a 

means of combining the benefits and power of standalone computing with the collaborative potential of the network 

in an easy-to-use distance learning environment. This ease of use and integrated functionality offers advantages to all 

distance students. IMMEDIATE’s ability to support students in remote locations beyond the reach of web-serverbased 

systems suggests the approach has real potential as a path towards bridging the digital divide in universitylevel 

distance education that merits further exploration. 

152

Technologically, the learning computer is distinguishable from either a LMS or a LCMS by its distributed 

functionality, asynchronous communications and integrated interface. But its most significant difference is that the 

distance students themselves, rather than the education providers, are the primary target users. An integrated, holistic 

approach is taken to e-learning in which the usability of the student’s overall computing environment is considered 

as important to successful learning outcomes as the learning content itself. 

The IMMEDIATE prototype is first and foremost a specialised learning environment in which network and 

education provider features are designed as support services rather than as authoring, teaching or course 

administration productivity tools. It demonstrates the potential for specialised computing environments to address 

usability concerns resulting from the increasing complexity of general purpose systems as they encompass more and 

more functions and features. 

IMMEDIATE models the forms in which learning will be delivered, while leaving most content to be authored 

separately or picked up from reusable learning repositories. The learning interface is assembled from reusable 

learning components, an extension of the drag and drop components of visual programming tools like Delphi or 

Visual Studio. Learning components differ from reusable software components (controls) in that they encapsulate a 

specific learning domain task rather than a more generally-applicable software feature. They differ from reusable 

learning objects in that they implement the form in which the content will be displayed rather than the content of a 

learning module. The learning components are the centrepiece of a modular construction which supports reuse on 

three levels – the student (multiple courses), the teacher (reusable learning materials), and the software engineer 

(reusable code). 

By combining an emphasis on interface design − especially user-centred adaptable and collaborative strategies − 

with an emphasis on e-learning as an extension of the human teacher and the university, the framework for an 

integrated, individualised, multi-dimensional, easy-to-use learning environment has been laid. This supports the 

findings of Murray et al. (2000) that “good interface design and passive but powerful user features can sometimes 

provide the benefits that are ascribed to more sophisticated or intelligent features” (para. 41). 

The learning computer concept is also distinct in that it draws upon the ubiquitous (embedded) computing approach 

to e-learning (Bork, 2001; Hoppe et al, 2000). The object is to so seamlessly embed the computer functionality into 

the learning domain – by taking into account the total context including hardware, software and human factors – that 

the computer will vanish for the learner. We are trying to enhance and extend the university learning environment 

rather than create an alternative virtual one. 

The Learning Shell uses specialised software to embed the computer in the learning environment in the form of a 

graphical user shell over the most widely-used family of operating systems, Microsoft Windows. In this way we 

implement a “what” interface centred upon the task in the user’s domain rather than a “how” interface focussed 

upon the computing mechanisms for achieving that task. This addresses a long-recognised challenge for simplifying 

and improving the usability of computer systems, as discussed in Gentner and Grudin (1990). The next challenge is 

to extend this approach to support special-purpose hardware, cross-platform functionality and mobile computing. 

Ongoing Work 

Research is proceeding along two axes: evaluating IMMEDIATE in an actual distance learning course, and reengineering 

the prototype to support mobile learning and low-cost hardware. 

Preparations are underway to trial a further-refined learning computer in the distance teaching at the university level 

of Maori and English as second languages. The trials provide opportunities to evaluate its pedagogical effectiveness 

with a particular emphasis on assessing the learning support and search facilities. They also provide opportunities for 

refining and testing IMMEDIATE's course authoring and management tool. 

IMMEDIATE implements the learning computer as an application that is installed on top of the Windows operating 

system on a specific computer. However, its anybody, anywhere, anytime goal would be better met if students were 

not confined to using their own computer but could use any available one. As a first step in this direction we are re- 

153

engineering the Learning Shell so it can be installed on and run from a plug-in flash-memory device as a mobile 

learning computer. 

Another possibility is to re-engineer the Learning Shell to run as a specialised user shell on top of a Linux kernel. 

While a Linux version offers the advantages of a cleaner and leaner implementation with lower system speed, 

memory and storage requirements, it carries a much greater programming overhead. All the rich interface 

functionality of the learning computer would have to be built from much more basic building blocks than in a 

Windows development environment like Delphi. As a medium-term goal, we are exploring the feasibility of 

implementing a Linux version along two lines: 

• As a low-cost single-purpose computer. Potentially, a Linux-based learning computer could be built from 

recycled PCs and components and distributed cheaply to students. Around the world millions of computers are 

scrapped as obsolete each year, many of which could be reclaimed for this purpose (ERG, 2006). 

• As a self-booting, cross-platform, portable device that can be plugged into any PC to temporarily convert it into 

a specialised learning computer. This offers a way for a learner to take full advantage of the learning computer 

approach, while sharing a computer with other household members and other tasks, or while studying from 

different locations. Black Dog (LinuxDevices, 2005) is an example of a portable self-booting device for running 

Linux-based software in a Windows environment. 

Reaching out to the many millions of potential e-learners beyond the ambit of current web technology is one of the 

biggest challenges and opportunities in computing today. 

References 

Attewell, J. (2004). Mobile technologies and learning. A technology update and m-learning project summary, 

London: Learning and Skills Development Agency. 

Bindé, J. (2005). Towards Knowledge Societies, Paris: UNESCO Publishing. 

Bork, A. (2001). Tutorial Learning for the New Century. Journal of Science Education and Technology, 10 (1), 57- 

71. 

Brandon Hall (2005). LMSs and LCMSs Demystified, retrieved October 15, 2007, from http://www.cedmaeurope.org/newsletter%20articles/Brandon%20Hall/LMSs%20and%20LCMSs%20Demystified.pdf. 

Comer, D. E. (1999). Computer Networks and Internets (2 nd Ed.), New Jersey: Prentice Hall. 

Dietinger, T., & Maurer, H. (1998). GENTLE - (GEneral Networked Training and Learning Environment). In T. 

Ottmann, T. & I Tomek (Eds.), ED--Media/ED--Telecom`98, Charlottesville, VA: AACE, 358-364. 

Dix, A., Finlay, J., Abowd, G., & Beale R. (1993). Human Computer Interaction, Prentice Hall. 

Education Review (2004). UK e-university ”fiasco”. Education Review, March 31-April 6, 2004, reprinted in EXMSS 

Off Campus, June 2004. 

ERG (2006). Electronic Waste, E-waste Research Group, Griffith University, Queensland, retrieved October 15, 

2007, from http://www.griffith.edu.au/engineering-information-technology/electronic-waste. 

Gentner D. R., & Grudin, J. (1990). Why Engineers (Sometimes) Create Bad Interfaces. In Carrasco Chew, J. & 

Whiteside, J. (Eds.), CHI'90 Proceedings, New York: ACM, 277-282. 

Goldberg, M. (1997). Communication and Collaboration Tools in WebCT. Paper presented at the Conference on 

Enabling Network-Based Learning, May 28-30, 1997, Espoo, Finland. 

Hope, W. (2002). Evidence of Unequal Access. NZ Infotech, 530. 

154

Hoppe, U., Lingnau, A., Machado, I., Paiva, A., Prada, R., & Tewissen, F. (2000). Supporting Collaborative 

Activities in Computer Integrated Classrooms – the NIMIS Approach. Paper presented at the 6 th International 

Workshop on Groupware, 18-20 October, 2000, Madeira, Portugal. 

IHug (2004). High Speed Satellite Internet, retrieved October 15, 2007, from http://www.ihug.co.nz/. 

Kubarek, D. (1999). Introducing and Supporting a Web Course Management Tool. Syllabus magazine, June, 

retrieved October 15, 2007, from http://www.cit.cornell.edu/atc/cst/SyllabusWeb/syllabus.pdf. 

Jesshope, C., Heinrich, E., & Kinshuk (2000). Technology Integrated Learning Environments for Education at a 

Distance. Paper presented at the DEANZ 2000 Conference, April 26-29, 2000, Dunedin, New Zealand, retrieved 

October 15, 2007, from http://www.deanz.org.nz/docs/jesshope.doc. 

Johnson, R., Kemp, E., Kemp, R., & Blakey, P. (2002). From electronic textbook to multidimensional learning 

environment: overcoming the loneliness of the distance learner, In Kinshuk, Lewis, R., Akahori, K., Kemp, R., 

Okamoto, T., Henderson, L. & Lee, C.H. (Eds.), Proceedings of 2002 International Conference on Computers in 

Education, Los Alamitos, CA: IEEE CS Press, 632–636. 

Jong, B.S., Lin, T.W., Chan, T.Y., & Wu, Y.L. (2003). Using VR Technology to support the Formation of 

Cooperative Learning Groups. In Devedzic, V., Spector, J. M.,Sampson, D. & Kinshuk (Eds.), Proceedings of the 3 rd 

IEEE International Conference on Advanced Learning Technologies, Los Alamitos, CA: IEEE CS Press, 37-41. 

LinuxDevices (2005). Pocketable Linux server creates plug-and-go Linux desktop, retrieved October 15, 2007, from 

http://www.linuxdevices.com/news/NS8562564746.html. 

Marine, S., & Blanchard, J-M. (2004). Bridging The Digital Divide: An Opportunity For Growth In The 21st 

Century, retrieved October 15, 2007, from http://www1.alcatel-lucent.com/doctypes/articlepaperlibrary/pdf/ 

ATR2004Q3/S0408-Bridging_opportunity-EN.pdf. 

Murray, T., Condit, C., Piemonte, J., Shen, T., & Khan, S. (2000). Evaluating the Need for Intelligence in an 

Adaptive Hypermedia System. Lecture Notes in Computer Science, 1839, 373-382. 

Patton, M. (1990). Qualitative Evaluation Methods, California: Sage Publications. 

Scott, C., Clayton, J., & Gibson, E. (1991). A Practical Guide to Knowledge Acquisition, Reading: Addison-Wesley. 

Smulders, D. (2003). Designing for Learners, Designing for Users, eLearn Magazine, February 3, retrieved October 

15, 2007, from http://www.elearnmag.org/subpage.cfm?section=best_practices&article=11-1. 

Yin, R. (1984). Case Study Research: Design & Methods (1 st Ed.), London: Sage Publications. 

155

Olfos, R., & Zulantay, H. (2007). Reliability and Validity of Authentic Assessment in a Web Based Course. Educational 


Reliability and Validity of Authentic Assessment in a Web Based Course 

Raimundo Olfos 

Pontificia Universidad Católica de Valparaíso. Mathematic Institute, Valparaíso, Chile// Raimundo.olfos@userena.cl 

Hildaura Zulantay 

Universidad de La Serena. Mathematic Departemant, La Serena, Chile // hzulantay@yahoo.com 

ABSTRACT 

Web-based courses are promising in that they are effective and have the possibility of their instructional design 

being improved over time. However, the assessments of said courses are criticized in terms of their validity. 

This paper is an exploratory case study regarding the validity of the assessment system used in a semi presential 

web-based course. The course uses an authentic assessment system that includes online forums, online tests, 

self-evaluations, and the assessment of e-learner processes and products by a tutor, peers, and an expert. The 

validity of the system was checked using internal and external criteria. The results show that the authentic 

assessment system addresses technical problems, especially regarding reliability of instruments. Some 

suggestions are proposed to strengthen authentic assessment in web-based courses and to test it. 

Keywords 

Authentic assessment, Reliability and validity, Web based instruction, Evaluation 

Introduction 

Web-based instruction offers flexibility, and it is becoming increasingly popular (Walles, 2002). Some experiences 

show Web-based instruction is as effective as classroom instruction (White, 1999; Ryan, 2001; Tucker, 2000). 

Moreover, Wilkerson and Elkins, (2000) concluded that students felt that web-based instruction was as effective as 

that in traditional courses. The use of the Web for instruction is in an early stage of development, and its 

effectiveness may not yet be fully known (Downs et al., 1999). In an extensive study including 47 assessments of 

web-based courses, Olson and Wisher (2002) assert that web-based instruction appears to be an improvement over 

conventional classroom instruction. 

To Phipps and Merisotis (1999) quality of the research in distance learning courses is questionable: much of the 

research does not control for extraneous variables; the validity and reliability of the instruments used to measure 

student outcomes and attitudes are questionable; and many studies do not adequately control for the feelings and 

attitudes of students. Attitude is a learned predisposition to consistently respond to a given social object (Segarra et 

al. 1997); for instance, technology tools. 

Authentic Assessment 

The use of new technology raises issues related to pedagogy, content, and interaction. As these issues are addressed, 

there needs to be a subsequent alteration in the type of assessment used on such courses and the associated 

procedures (Walles, 2002). 

A new approach to evaluation is authentic assessment. This modality connects teaching to realistic and complex 

situations and contexts. Also called performance assessment, appropriate assessment, alternative assessment, or 

direct assessment; authentic assessment includes a variety of techniques such as written products, portfolios, check 

lists, teacher observations, and group projects. 

According to Herrington and Herrington (1998) authentic assessment occurs within the context of an authentic 

activity with complex challenges, and centers on an active learner that produces refined results or products, and is 

associated with multiple learning indicators. It includes the development of tests and projects (Condemarín, 2000). 

The AAS not only evaluates the products, but also the processes involved. It is a process that monitors the learner’s 

progress using a variety of methods, such as observation records, interviews, and evidence gathering. It is consistent 






156

with Vygotsky’s dynamic assessment in that it is a mediated process in which social interaction stimulates the 

learning process. It also allows for collaborative learning. 

Authentic assessment of educational achievement directly measures actual performance in the subject area. It was 

developed as a result of criticism of multiple-choice tests, which usually only provide a superficial idea of what a 

student has learned and do not indicate what a student can do with what was acquired (Aiken, 1996). Authentic 

assessment can provide genuine accountability. All forms of authentic assessment can be summarized numerically, 

or put on a scale, to make it possible to combine individual results and to meet state and federal requirements for 

comparable quantitative data (National Center for Fair and Open Testing, 1992). One method to this end, according 

to Wiggins (1998), is the use of rubrics, which are sets of criteria that evaluate performance. Points are assigned 

according to how well each criterion is fulfilled, and are then used to provide the quantitative values. 

The validity of Authentic Assessment 

Aiken (1996) plainly indicates the difficulty in establishing the validity and reliability of any authentic assessment. 

Schurr (1999) states that a disadvantage of authentic assessment is the difficulty, maybe even the impossibility, of 

obtaining consistency, objectivity, and/or standardization in its results. Wolf (cited by Herrington and Herrington, 

1998), sees the problem as a tradeoff between validity and reliability. In the same vein, Linn, Baker and Dunbar 

(1991), hold that performance-based assessments are valid in terms of consequence, impartiality, transference, 

content coverage, cognitive complexity, significance, judgment, cost, and efficiency; consequently, reliability, 

understood as the stability of the results, is difficult to obtain. 

Linn et al. state that “a greater dependence of critical judgments on the performance of tasks” is inevitable. This is a 

problem for large-scale assessments, but not for the smaller and more specific contexts to be found in superior 

education (Gipps, 1995), or in those cases in which the student is evaluated through class activities, where learning 

and evaluation are essentially one and the same (Reeves and Okey, 1996). 

There are also problems with reliability in standardized tests. Gipps (1995) mentions national tests in the UK with 

reliability rates in posttests that are lower than those obtained in performance tests. Reeves and Okay (196) state that 

authentic assessment takes place within a real-world context, where generalizations are of little value and, therefore, 

reproducibility should not be a concern. Young (1995) adds that assessment needs can be perceived in a more 

functional way, and assessments can be validated in terms of their value in the real world rather than for their 

stability as instruments (Herrington and Herrington, 1998). 

Authentic assessment in web based courses 

Computing-mediated distance education introduces extraneous factors that could affect the validity of the course 

assessment system. One of these factors is identified as the usability of the web site. Usability deals with how well a 

system satisfies user needs and requirements. It applies to all the aspects of a system with which a user might 

interact, including installation and maintenance procedures. It is usually associated with aspects such as ease of 

learning, user friendliness, ease of memorizing, minimal errors, and user satisfaction. (Sánchez, 1999, 2000). 

Another factor is attitude. According to Ceballos et al. (1998), only when there is a favorable attitude toward the 

TICs, can an e-learner effectively face learning tasks; therefore a web-based assessment system requires a positive 

attitude from the users to show their full potential. In other words, the degree of effectiveness of the assessment 

system could be affected by a negative attitude towards the TICS by some of the e-learners. 

Literature provides scarce information about authentic assessment in web-based courses. It only refers to case studies 

and similarities. Weller (2002) examines technical barriers in the assessment process of a web-based course, and 

points out the tension between individuality and robustness in submissions and the detection of plagiarism. Clarke et 

al. (2004) state that feedback to students on their assignments is an essential activity. They point out that tutorial 

support must be a crucial part of good distance course, where emails should be considered as a non-intrusive means 

of communication. Orde (2001) offers some suggestions to develop online courses, for instance: to consider a 

description of learners; to provide readily available technical support; to eliminate group activities easily done faceto-face; 

and to record and grade interactions such as e-mail and group discussion contributions. To Orde, the testing 

157

portion of CourseInfo requires that each quiz item be individually entered and submitted. If this feature of the 

software is used, the ID students advise allowing multiple attempts. To Orde, formative evaluation is an essential 

component and necessary to online course development. 

Gatlin and Jacob (2002) discuss advantages of digital portfolios as part of one university’s authentic pre-service 

teacher assessment. Chang (2002) asserts that web based learning portfolios are useful for students to obtain 

feedback from other students. Collins et al (2001) state that the design of assessment for web based courses should 

measure student participation, and the development of skills. 

Literature recommends centering authentic assessment on individual tasks, but also connecting it to real life, 

including interactive work among peers. Fenwick and Parsons (1997) assert that effective assessment must be 

intricately woven throughout the teaching-learning process. Collaborative learning activities enable subjects to share 

their abilities and limitations, providing a better quality product than one that is the mere sum of individual 

contributions. Group interactions, following the indications for collaborative work given in a course, facilitate 

vicarious learning, which is hard for some subjects to experience if they do not interact with their peers (Bandura and 

Walters, 1963; Vygotsky, 1985). 

The purpose of this study 

The purpose of this study is to analyze the validity of the Authentic Assessment System (AAS) used in a web-based 

online course. Leading questions of this study are: Is there consistency between AAS results and effectiveness of 

products in a real life context? How much do AAS results correlate with external individual learning criteria? What 

evidence can be gathered about the reliability and validity of AAS’s different phases? “Do external factors such as 

web usability and participants’ attitude invalidate the AAS design? 

Method 

Subjects 

Twenty-eight teachers participated, 13 primary school teachers and 15 secondary school math teachers, who taught 

different levels of math, from 5th to 10th grade. All except one of the participants worked in schools in two nearby 

cities or their surroundings. The cities had a population of approximately 120,000 inhabitants each. None of the 

students had previous experience with distance learning. 

Setting 

A.- The course. This study took place within the context of the twelve-week course called "Didactics of Elementary 

Algebra" with 89% effectiveness, offered to mathematics teachers. Twenty-five of the 28 teachers –hereinafter called 

students- passed the course and met AAS requirements. It was a semi presential course. The students did most of the 

activities by means of distance learning. To do so, the students had access to a web site with the course contents and 

computer resources such as e-mail, forums, FTP, and online tests in order to interact with the rest of the students and 

with the tutor in order to meet the course's evaluation requirements. The course included three classroom activities 

and an authentic assessment system made up of eight internal examinations. 

The course was managed by a coordinator and led by a tutor, both of whom were academicians with experience in 

distance learning. The students used computers as a means of communication with their peers as well as with the 

tutor. In addition, computers were used to do the course assignments as well as facilitating the search for and 

creation of didactic material. 

The course objectives were: a) to develop the capacity to design, implement, and evaluate didactic units regarding 

elementary algebra, b) strengthen the capacity to work collaboratively, design and evaluate the use of didactic units 

centered on elementary algebra, and c) develop Internet skills with the aim of strengthening professional skills and 

efficiency in teaching elementary algebra. 

158

The first unit, "Internet tools" (1 week), included a classroom session in which surfing the Internet, using e-mail, and 

the website's communications services were reviewed. The second unit, "Didactic Design Concept" (2 weeks), 

showed the students how to characterize the unit's terms, didactic design, lessons and activities, by establishing a 

basic language and conceptual framework upon which to develop the didactic designs. An individual evaluation was 

included. It was called "Design forum" (I-1) referring to units 1 and 2. The third unit, "Professional Knowledge" (2 

weeks), introduced specific concepts on types of learning, curricular conceptions, and learning evaluation. The unit 

included an individual examination called "On-line test" (I-2) referring to course units 1, 2, and 3. 

The fourth unit, "Creating a Didactic Design" (4 weeks), guided the student in creating a didactic design. In this unit, 

collaborative work in small groups was realized, in which decisions were negotiated and roles were assigned to the 

participants with the aim of achieving common objectives. Three evaluations were done: assignment of “individual 

contributions" (I-3), the group creation of a "preliminary didactic design" (I-4), and the creation of a "didactic 

design" (I-5). During this period, the fifth week, a classroom session took place to get feedback from the 

collaborative work. Material was shared and the work was checked ahead of time. The fifth unit "Validation of the 

Designs" (2 weeks) provided the guidelines for testing the didactic designs with schoolchildren. In addition, it 

provided the guidelines for creating a final report that included a commented version of the didactic design 

subsequent to its application. This unit included three evaluations: the "final report" (I-6), a self-evaluation of the 

design through the "Own Design Forum" (I-7) and self-evaluation or "personal evaluation guideline" (I-8). The 

course ended after 12 weeks with the third classroom session. In this session, the designs were shared. 

B.- Authentic Assessment System. The assessment system regarding the students' learning and group production 

included the eight examinations mentioned (I-1 to I-8), which were structured following the four criteria summarized 

by Herrington and Herrington (1998). 

• The authentic assessment took place in a real context. It was connected with the students' daily professional 

performance (Meyer, 1992; Wiggins, 1993; Reeves & Okey, 1996). 

• In the authentic assessment, the students assumed a leading role in collaboration with their peers (Linn et al., 

1991; Kroll, Masinglia y Mau, 1992) displaying their performance and presenting their products (Wiggins, 1989, 

1990, 1993). 

• In the authentic assessment, the evaluation activities were authentic. In other words, they were inextricably 

integrated with course learning activities and the student's daily professional activity (Young, 1995; Reeves & 

Okey, 1996). The evaluation activities corresponded to unstructured, complex challenges which demand 

students' own opinions (Wiggins, 1993; Linn et al., 1991; Torrance, 1995). 

• In the authentic assessment, multiple indicators (Lajoie, 1991; Linn et al., 1991) and several criteria were used to 

grade the variety of requested products (Wiggins, 1993; Lajoie, 1991; Resnick & Resnick, 1992). 

Table 1. Summarizes structure and content of each instrument used in an AAS 

Description 

Procedure 

I-1 Forum Design 

Participation in the forum, Dichotomist (does/ does not) 

I-2 On Line Test 

30 graded items attending 6 content dimensions, 3 levels of complexity.(Resent 

options were admitted) 

I-3 Individual Contributions 2 records referred to e-mails sent Dichotomist:Sent pertinent messages or not 

I-4 Preliminary Design 5 graded records referred to pre design attributes. Rubric based 

I-5 Didactic Design 

11 graded items referred to Design. Rubric based 

I-6 Final Report 

9 graded items referring to the design application. Rubric based 

I-7 Own Design Forum 1 numeric record summarizing didactic design quality opinion 

I-8 Personal Evaluation 20 Likert scale items. Knowledge, abilities and feelings acquired 

Individuals were basically tested on knowledge domain. The first part of the course provided instrumental 

knowledge about how to build didactic design, how to work in a collaborative approach, and how to use e-mails and 

the web site. Individuals were required to answer various tests and were also asked to report their learning perception 

during the process. The second part of the course focused on a productive approach. Participants formed small 

groups to elaborate didactic designs. Accordingly, evaluation procedures were applied to partial products of the 

didactic design building. 

159

Table 2. Assignment evaluation responsibilities and the weight of each procedure in relation to the assessment 

system 

Responsible of assignment Evaluated Weight 

I-1 Automatic assignment 

I-2 Automatic assignment. 

I-3 Automatic assignment. Pertinence was decided by tutor 

I-4 Tutor does assignments. 

I-5 Peers: two or more colleges do assignments 

I-6 Expert does assignments 

I-7 Assignment by participants (self evaluation) 

I-8 Self evaluation learning 

Research Design 

Individual 

Individual 

Individual 

Group Group 

Group Group 

Individual 

This single case study focused on multiple sources as Yin (1984), Stake (1995) and Weiss (1997) have 

recommended. Tests, archival records, and participants’ perspective were recorded. According to Yin (1994), this 

case study design was used to describe the AAS implemented in a specific web based course, and to analyze several 

variables related to a small number of subjects. 

The study considered individuals and small groups as units of analysis, according to an embedded case study design 

(Yin, 1994). Data were connected to indicators by means of correlations and triangulation analysis. Traditional alpha 

0.5 and 0.01 were used. 

Both the evidence of validity based on criteria of judges, parallel instruments, and non obstructive data, and the 

evidence of reliability obtained from indices of association between equivalent items, and reiterated measurements, 

are sustained in correlations. In effect, the Rho of Spearman coefficient for nonparametric data, index r of Pearson 

for variables of normal distribution, and the value alpha of Cronbach for groups of variables, express a correlation or 

degree of association between the measured variables. These univariated measurements are used in exploratory 

studies because they are simple and they aid in elaborating integrated models. 5% and 1% of the critical values 

considered in this study are used habitually in education with cuasi-experimental designs, unlike the values such as 

0.1%, or lower, used in experimental designs or in the area of medicine. 

Instruments 

This study used records of the eight AAS procedures, and three additional data sources to validate the AAS. 

The eight AAS instruments 

AAS was made up of 8 procedures associated with their respective instruments. When using the Cronbach 

coefficient as an indicator of AAS's reliability, we got a positive, but not significant value (α(26)=.22, p>.05), which 

was expected because AAS measurements did not refer to a one-dimensional variable. 

Internal Instruments 

Table 3. Correlation of each instrument with the rest of the AAS 

Correlation 1 (one-tailed) 

I-1 Forum Design contribution 

rho(26) = 0,37 p.05 

I-3 Individual Contributions mailed 

rho(28) = 0,50 p.05 

I-7Own Design Forum opinion 

rho(26) = 0.47 p

The eight AAS procedures measured different types of individual learning and group work. The eight correlations 

obtained between each instrument and the rest of the assessment system, as internal consistence indicators, were low, 

making it evident that the measurements were heterogeneous. See table 3. 

The online test (I-2) reached Cronbach α(26)=.88, which was used as a reliability measure. The Likert scale (I-8) 

reached Cronbach α(13)=.96, p

E-3c. Likert scale on usability: The scale was implemented as an indicator of the incidence of the usability of the 

web site in the course results. An adaptation of the Sanchez (2000) usability test was carried out, focusing on the 

measurement of the usability of the Website in relation to the course’s authentic assessment system. The author used 

five proposed dimensions: learning, satisfaction, error, efficiency and memory factors. Twenty-three items were 

created. They were assigned a score on a Likert scale, of 1 to 5 points, one being the lowest grade and five being the 

highest. The test was subjected to the remote criticism of educational research specialists for the validation of its 

content. The instrument's internal consistency was measured with Cronbach's alpha, obtaining respectively for each 

one of the factors, the coefficients 0.68, 0.77, 0.84, 0.54, 0.89, and for the complete instrument, α = 0.74 (n=22). 

Procedures 

The first two procedures 

They refer to global indicators of AAS’ validation: the relation with real context consequences, and with individual 

learning. 

Consistency between AAS’ results, and product effectiveness in real context. After designs were elaborated, materials 

for schoolchildren were reproduced to use in one or more classes. Then lessons were implemented and 

schoolchildren were evaluated. The averages of these evaluations were compared with the students’ AAS results by 

means of Spearman’s range correlation coefficient. 

Measurement of degree of association between AAS results and individual learning criteria. A post-test was 

considered as an indicator of the students’ learning. The posttest (E-2) was applied at the end of the course without 

prior warning; the students had not prepared for the test, and the results did not have any effect on the class grades. 

For this concurrent validity analysis, the students' AAS average was correlated with the post-test average. 

What evidence can be gathered about AAS validity in either individual or group phases? 

Analysis of the validity of AAS’s Individual Component 

The individual phase focused on instrumental learning. Three criteria were considered:. 

The student's capacity to recognize the components of a didactic design. The student's capacity to recognize the 

components of a didactic design was measured in one of AAS's instruments as well as in the external post-test. This 

analysis compared the percentages of correct answers from the items referring to this capacity included on the 

posttest (E-2) as well as on the online test (I-2). 

The student's capacity to surf the web site and send e-mails. This analysis about the students' capacity to navigate the 

web site and use e-mails was based on information gathered from the evaluations applied to the students and from 

non-obstructive information from the website. Specifically, two indicators of the students' capacity to surf the 

website and use e-mails were created. The first indicator was the sum of four AAS data, namely: a publication in 

"forum 1" (I-1), answer to an item in the "on-line test" (I-2), answers to two items in the "self-evaluation" (I-8). The 

second indicator was the sum of the data obtained from the instruments that were external to AAS, namely: the 

number of e-mails sent, with a maximum of three (E-1 b), an item from the “external questionnaire” (E-3 a), three 

items referring to the website usability (E-3 c), and three items from the post-test" (E-2). To create these indicators, 

each variable was adjusted to a scale of 0 to 1, without considering the values assigned to the AAS instruments. The 

AAS information was correlated with the indicator associated with external data. 

Students' opinion on learning about didactic designs and IT. During the course, there were two opinion instruments 

that referred to the learning level attained during the course; One was part of the AAS (I-8) and the other was not (E- 

3 a). Although the instruments were administered at different times, one in the middle and the other at the end of the 

course, the degree of association between the answers of groups of equivalent items was used as an indicator of the 

criterion's validity. One indicator compared "the opinions on learning with didactic designs" which were made 

162

during the external tests as well as during the internal tests which were part of the AAS. Another indicator compared 

"the opinions made on both questionnaires regarding acquisition of computer skills.” 

Analysis of the validity of the AAS's Group Component 

The collaborative phase focused on group production. In this section, the validation procedures related to AAS along 

with collective information are described. 

Consistency among peer opinions on didactic designs. Once the groups created their didactic designs, the designs 

were submitted to two or three students that were not part of the group. Each one of these students independently 

evaluated his or her peers' work according to designed guidelines based on rubrics. The correlations between the 

evaluations of the different peers on the same design were used as a peer evaluation consistency index (I-5). The 

average of the indexes associated with evaluations of each one of the 8 designs created by the students was 

considered as the internal peer evaluation consistency index. 

Consistency between the AAS evaluations on creation and validation of the didactic designs. The analysis, in this 

case, refers to the four evaluations regarding the creation and validation of the didactic designs (I-4 to I-7). 

- Consistency between the tutor, peer, expert, and student evaluations on didactic designs. The first three evaluations 

(I-4, I-5 and I-6) were structured as part of a series of criteria defined by rubrics. Only the final evaluation, the selfevaluation 

(I-7), consisted in a grade, on a scale of 1 to 7, which corresponded to the student’s overall appreciation of 

the design created by his or her work group. 

The four evaluations, although they refer to the same object, i.e. to the didactic design, measured its diverse aspects 

at different times. Such measurements were made with different emphases, which were indicated by means of 

assigning different values to the items. As a criterion of consistency, the four valuations were correlated in two parts. 

- Consistency between the peer and the expert evaluation regarding the level of suitability of the activities for the 

learning expected to be achieved by means of the designs. The didactic designs were assessed by the peers before 

they were applied in the classroom. Then, after they were applied (I-5), they were assessed by an academic didactics 

expert (I-6). The peers as well as the expert provided their opinions regarding the "suitability of the activities 

proposed in the designs to achieve the purpose of the designs themselves". The opinions refer both to the learning 

activities as well as to the evaluation activities. The opinions were made within the context of a scale associated with 

rubrics, and they made up part of AAS’s instruments. In other words, the opinions made up part of the guidelines 

applied by the peers (I-5) and the guidelines applied by the academic didactics expert (I-6). 

Web usability and participants’ attitude as possible invalidation factors 

A significant positive correlation between AAS results and factors such as web usability and participant attitude was 

understood as an invalidation factor of the assessment system. 

Participants’ attitude. The Attitude towards mathematics Test (Aikeen, 1996) was adapted to measure student 

attitude to the course’s AAS. 24 items were organized according to a Likert scale. The test reliability was Cronbach 

α=0.62. 

Usability. The Sanchez usability test (2000) was adapted to measure the usability of the AAS component of the web 

site. The 21 items referring to five factors called learning, satisfaction, error, efficiency, and memory were organized 

according to the Likert scale. The test reliability was Cronbach α=0.74. 

Results 

This section provides the test results used to characterize the AAS’s validity and reliability. These tests mainly 

consisted in correlations as indicators of concurrent validity and coefficients of internal consistency; which were 

163

organized in four groups, namely: The tests that compared the AAS results with the student’s learning and with the 

group productions in a real context; the isolated tests as indicators of validity of the AAS’s individual component; 

the tests related to the validity of the AAS’s group component; and, finally, the tests related to two possible 

invalidation factors. 

AAS’ relation with both real context products and individual learning 

Consistency between AAS’ results and products’ effectiveness in real context. The averages of the children’s results 

were compared with the teachers’ AAS results. Two of the teachers’ reports about design application did not provide 

enough information to be considered in the correlation analysis. As shown in table 5, some designs were 

implemented in more than one class. The Spearman correlation was positive, rho(6)=.30 p=.2, but not significant. 

Number of classes 

Schoolchildren results 

AAS average results 

* missing value 

Table 5. Results of implemented designs 

Groups 

Gr1 Gr2 Gr3 Gr4 Gr5 Gr6 Gr7 Gr8 

3 2 2 1 4 m.v.* 3 m.v.* 

66% 80% 80% 98% 93% m.v.* 74% m.v.* 

70 74 78 77 92 53 79 80 

Measurement of degree of association between AAS results and individual learning criteria. The Spearman 

correlation as a concurrent validity test gave a value rho=.24, p>0.05, which was not significant. 

Validity of AAS's individual component 

The individual component was instrumental instruction, meaning that, in this part of the course, they were given 

conceptual tools to elaborate didactical designs in order to progress to a collaborative approach during the next part 

of the course. 

The students´ ability to recognize the components of a didactic design. The item “What are the main phases of a 

didactic design?” included on the pre-test, as well as on the online test (I-2) and on the post-test (E-3), had an 

unexpected variation regarding the number of correct answers in each of the applications: Meanpretest = 5/20 (25%), 

Meanon_line_test = 19/25 (76%), and Meanpost-test = 12/21 (52%). 

The online test and post-test results were expected to be consistent, particularly those items that were the same. 

Nevertheless, the correlations were not significant; they were low and even negative. The correlation was rho(19)=- 

.23, p>.05, (two-tailed). 

The students' capacity to explore the website and send e-mails. The estimated level of association between external 

information and that gathered through AAS according to a Spearman correlation was rho(23)=0.44, p

Validity of AAS´s productive evaluation component 

The group component was focused on a productive approach, where students were asked to build and apply a design. 

Peer evaluation (I-5). The peer evaluations of the same design were quite coincidental. A high average of internal 

consistency for an average of 8 groups was attained: α(6)=0.96, p

The correlations were not high enough to be significant. They were rho(5)=0.11, p>.05 and rho(6)=0.46, p>.05, 

respectively. As one can observe, both the expert and the peers assigned high scores to their evaluations. The slight 

differences between them led to null correlations. 

Web usability and participants’ attitude as possible factors of invalidation 

Attitude of participants. Students’ attitudes to the AAS aspects of the WEB were positive, and the association of this 

variable to AAS results was null, rho(20) = 0.09. 

Web usability. Students considered AAS’ aspects of the web usability to be adequate. The correlation between 

students’ evaluation of AAS aspects of web usability with their results in AAS was positive, but not significant. The 

Spearman correlation was rho(15)=0.27. This means there was no significant relation to affirm that the variables 

were directly related. 

Discussion 

This chapter summarizes the study results, analyzes the technical problems of the AAS, and offers suggestions to 

elaborate an Authentic Assessment System for a web-based course. 

Study results: Validity of AAS 

This study considered three perspectives to analyze the Authentic Assessment System’s validity. The first 

perspective, using concurrent validation techniques, analyzed the AAS’s validity as a whole, as a one-dimensional 

variable. The second perspective, also using correlation techniques, analyzed local aspects of the AAS, focusing on 

performance in real contexts and non-obstructive data. The third perspective was a “reciprocal” modality; instead of 

validating the AAS according to its degree of association with other criteria, the objective was to demonstrate a null 

association between the AAS and possible invalidation factors of the system. 

One-dimensional perspective. From this view, the AAS results were considered as whole values, which were 

correlated with two external variables. One was the degree of effectiveness of the didactic designs elaborated by the 

students, and the other was the students’ results in a posttest. The two correlation indexes were positive, but neither 

was significant; thereby confirming the difficulty in validating the authentic assessment procedures, as several 

researchers warned (Linn et al., 1991; Aiken, 1996; Herrington and Herrington, 1998; Schurr, 1999). The question 

remains if other indexes inform some degree of validity of the AAS used in the course. 

Desegregated perspective. Considering that the AAS was multidimensional and that diverse factors affected the 

results, this second approach separately analyzed the AAS’s group component from the individual component. These 

analyses were limited to selected data, and some gathered non-obstructively. Out of the five indicators used to 

analyze the individual component, four correlated significantly: the ability to search in the web, the ability to use email, 

the opinion about ICT learning, and the ability to build didactic designs. On the other hand, only three 

indicators used to analyze the AAS’s group component were significantly favorable, which can be partially 

explained by the small number of student groups. In fact, the third indicator, in spite of reaching the correlation rho 

= 0.46, was not significant. This second analysis perspective was fruitful, offering some insight in terms of the 

characteristics of an authentic assessment system that evidences validity. 

Reciprocal perspective. The third perspective of analysis of the AAS’s validity confirmed that we cannot attribute 

the students’ success to either of the two invalidation factors considered, the attitude towards the evaluation system 

and its usability. Under this perspective, the AAS offered an additional partial evidence of validity. 

Then, the results of this exploratory correlational case study indicated some evidence of validity, but also several 

technical problems with the authentic assessment system. We may conclude that the authentic assessment system 

implemented with two components, one individual and the other in groups, showed some problems and some 

insights. 

166

The following sections offer interpretations based on the in-depth analysis of available data. These interpretations 

give rise to suggestions for elaborating authentic assessment systems and to formulate hypotheses in subsequent 

studies. 

Reliability and validity of each AAS’s component 

AAS's individual component. It should be mentioned that the correlation between AAS’s results and the posttest 

achievement was not significant. Although AAS's results were good with 89% effectiveness, they cannot be 

understood as learning in the manner that the posttest measured it. Even though the post-test pre-test differences 

were significant according to the t-test (p

Data illustrate some reliability and validity problems, which are very typical of authentic assessment. Because 

effectiveness of products reached through the course referred to variables measured in context, it was difficult to 

obtain a good correlation with both the process and the results of the course. Similarly, since authentic assessment is 

related to complex learning in special multiple dimensions, it was also difficult to obtain a good correlation with a 

single multiple-choice test. 

We should mention that AAS instruments were technically designed to measure knowledge, abilities, competency, 

and attitudes in accordance with the main principles of authentic assessment. In addition, AAS has evolved into a 

highly effective course. So in spite of the evidence of lack of reliability, this assessment system shows evidence of 

cognitive complexity, significance, and efficiency; that is, authentic validity. 

Normal-based and criteria-based evaluation. According to an “edumetric” assessment model, almost all people are 

expected to learn, but according to the psychometric measurement model, normal distribution is expected to describe 

individual variability in learning. In this second view, learning differences between higher and lower groups are 

expected to be stable and significant. So, reliability is expected as a necessary condition of validity. According to the 

former view, significant differences are not expected between learners. So, some slight differences arising from 

uncontrolled variables are expected. Therefore, stability of differences in results should not be considered as the 

central point of an effective course validation. 

Straetmans et al., (2003) assert that principles of classical test theory may be applied to qualitative assessment, and 

measurements of quality control such as validity and reliability should be applied to forms of competency assessment 

in a similar way as they were used for other tests. We should keep in mind that psychometric refers to maximizing 

differences between individuals and edumetric refers to measuring within-individual growth, without reference to 

other individuals. Frederiksen & Collins (1989) definitively propose specific criteria, and criteria other than validity, 

for newer forms of assessment. 

Valid measures with low reliability. Several correlations used in this study refer to the consistent association between 

variables, in relation to reliability. Stability, consistency, and concurrent validity were not reached in data. How 

much stability should be expected as part of the validation process? Can valid measures with low reliability be 

obtained, as is often the case in authentic assessment systems and was this the case in this study? We understand that 

both reliability and validity are expected in an assessment process. But, following the traditional path, validity is 

sacrificed in order to obtain reliability, and according to an authentic assessment approach, soundness of 

measurement is privileged as compared to stability of results. 

Four major problems that affect AAS validation 

Online tests, rubrics measuring, self-evaluation techniques, student dropout, and restrictions in timing tasks, which 

are common in distance education, demonstrated some technical problems. In effect, the online tests with the 

possibility of repeating answers is questionable; the self-evaluations should be adjusted to more specific referents; 

the use of rubrics looks promising but needs to be improved; and the various participants’ contributions to the 

evaluation process, which has proven to be effective and widely accepted, still needs to be adjusted. 

Validity of the online test with multiple attempts. The opportunity granted to the students to answer the online test 

after several attempts did not favor AAS's reliability. The online test was part of the authentic assessment system's 

individual phase. In order to complete the test, the computer system allowed the students to make several attempts 

before recording their final answers. Even though this favored AAS performance, the online test was not confirmed 

as being a contributing factor to the improvement of post-test scores. Students seem to only partially retain the 

knowledge that was required of them on the posttest, which they had taken two months after taking the online test. 

The opportunity provided by the online test proved to be attractive to the students. Once the test was completed, the 

computer system automatically sent them the test results with percentages for each group of questions. Since these 

percentages referred to groups of items and not each individual item, and the computer system assigned partial credit 

to the partially correct answers, it was not easy for the students to know which items they had gotten wrong. They 

needed to analyze each item of a group in order to improve their test scores. 

168

This opportunity may have had a positive short-term effect without requiring long-term retention of the material. The 

knowledge being tested on the online test was part of the theoretical knowledge that the students would use when 

creating their didactic designs. Therefore, we expected that this theoretical knowledge would be meaningful to them, 

thus remaining in their long-term memory until taking the posttest. Apparently thought processes during the on line 

test were not sufficient to consolidate such knowledge. 

Validity of AAS's group component: This component included tutor, peer, expert and student evaluations. Only 

partial evidence of the consistency between the tutor, peer, expert, and student evaluations regarding the creation and 

validation of the didactic designs was obtained. The peer and expert evaluations agreed that the didactic designs 

were good. However, they were not coincidental enough when establishing a hierarchy with regard to that quality. 

Although the evaluations carried out by the different participants were positive and the didactic designs in question 

were effective, the evaluations were not consistent enough among themselves. Why was there not enough 

consistency between the peer evaluations and the expert evaluation in the cases when they assessed the same aspect? 

The dissonance could have its origins in the assessment instruments' deficiencies and in the research design's 

insufficiency. This would not have allowed the assimilation of the fact that the students' behavior varied over time. 

The group component evaluation that was not carried out on the basis of rubrics was a self-evaluation of the students. 

The assessment consisted of grading the didactic design on which they worked. Apparently, it is necessary to check 

the effect of subjectivity on the self-evaluations. We suggest doing this because the students evaluated their designs 

quite positively. On average, they assigned themselves a score of 97%. This differed from the expert’s evaluations 

of the reports on didactic designs, which averaged only 75%. The objectivity might be increased if the students also 

assigned themselves a grade on the basis of rubrics and not based on holistic perception. 

Dropouts. Another element to consider is the number of groups about which the data are gathered. The low number 

of only eight work groups constituted a limitation on the analysis of AAS’s group component. Statistical significance 

is difficult to obtain with a limited number of cases. In addition to following up on evaluation procedures and the 

methodological design adopted to examine AAS validity, an analysis of the students' behavior should be conducted. 

Not all of the students finished the course as expected. Only 52% (13 of 25) of the students submitted their selfevaluations 

despite the fact that it had a detrimental effect on their grade. Only five of the eight groups submitted a 

completed final report. In spite of these deficiencies, the students were able to pass the course with a good grade. 

The students would have liked to have finished the course by submitting all of their work. However, the due dates 

expired and, faced with high demands at their jobs, they did not meet the final requirement. 

Mortality, referring to subjects who were not included in the analysis because they did not provide certain responses, 

had an important effect on the study. In fact, if the omitted answers had been considered incorrect and their 

corresponding instruments had received a score of zero, not only would we have obtained positive correlations, but 

also the majority of them would have been significant. This would have corroborated the validity of the AAS. 

Time factor limits production. During the creation of the didactic designs, the evaluators indicated the weak aspects 

of each didactic design. This gave the students the opportunity to correct their designs before the next evaluator's 

revision. This allowed, within a highly limited time frame, time margins for some groups of students to improve their 

weakest aspects. Thus, the designs had the tendency to improve. However, the groups' work could not always be 

completed within the established period of time. This was noticeable at the end of the course, when the expert 

conducted the final evaluation of the didactic designs subsequent to their application in the classroom. The moment 

arrived when the students had to deliver their reports in whatever state they were in, and in some cases this implied 

an interruption in their production. 

Final suggestions to improve evaluation systems in web based courses 

We expect improving assessment instruments and the assessment system design of a web based course by means of 

successive approximation processes: Multidimensional authentic assessment demands, integrated techniques, 

purified instruments, and an extensive sample to achieve acceptable reliability. 

Refining and integrating evaluation instruments. Using integrated evaluation procedures that pervade the course's 

main concept seems to be appropriate. Considering that the authentic assessment system is multidimensional by 

169

nature, some specific core objectives would be identified, and the number of variables to be assessed should be 

reduced. Thus, the evaluation process would be permanently focused on those objectives with the aim of monitoring 

progress.. 

Bearing in mind that the results of the self-evaluations and the evaluation through rubrics were very elevated, the 

question arises whether those results came from low requirement levels. In order to supply a satisfactory answer, 

standardized evaluation instruments are vital, which unfortunately stray from the authentic assessment method. 

Self-regulation techniques such as self and co-evaluation proved to be attractive, because they fostered course 

effectiveness. However, they need to be refined in order to attain better reliability indices and evidence of validity. 

Self-regulation techniques seem to be a high priority; experts establish more rigorous criteria on the quality of the 

products to be created by students, safeguarding AAS’s ecological validity. In a course for teachers, didactic designs 

effective in the classroom should be supported on the basis of testing the schoolchildren on whom the designs are 

applied, with validity recognized by peers as well as by the expert. AAS should provide the guarantee that classroom 

achievements are adjusted to anticipated learning according to the official attainments. 

Even though pertinence and effectiveness of didactic designs are essential requirements for favoring the assessment 

system validity, they are difficult to meet. 

In this study, the validation of AAS's group component provided evidence that the procedures and instruments 

employed require greater integration, as the literature recommends (Chong, 1998; Roschelle, 1992; Lave, 1991). 

Improving Rubrics: Rubrics need to be further developed. Their use requires a great deal of specificity in their 

operation. Perhaps it is appropriate for the evaluators to limit themselves to areas of specific domain. An example 

would be for the same evaluator to judge the same aspect in several didactic designs instead of judging all of the 

dimensions of a specific design. 

Improving the authentic assessment design. When assessment design includes both individual and group 

components, a differentiated evaluation could be instated within each group on the basis of individual roles. The 

students could assume differentiated tasks that are assessed individually even though the final product is a shared 

responsibility. For example, a student can act as a graphic designer and learn more than his or her peers about the 

use of related software. In each subject, individual growth, according to previous experience, expectations, and 

variety of interactions generated within the group, can be verified. In this manner, we assume a priori that the types 

of learning are different. However, at the same time, they support each other and enrich the collaborative work's 

productivity. 

Even though this approach would be far from the traditional approach that measures all students with the same stick, 

it would be consistent with the principles of an authentic assessment and would favor AAS's ecological validity. 

A research design that takes recommended variations into account should be assimilated into a time series analysis. 

The data should be gathered gradually throughout the process so that progress may be observed and consistency 

between measurements may be appreciated. Within that context, information should be gathered about those aspects 

of the students and their work in which change is expected, which at the same time provides feedback to the 

individual learning and group work processes. 

Qualitative data support 

Qualitative data were gathered during the AAS application process. Content of students’ e-mails, an open-ended 

questionnaire, two focus group discussions, and notes from participant observation strengthened our understanding 

of the AAS limitations. A qualitative approach was not developed in these pages because of its length; however some 

commentaries were provided that appear in the discussion section. These comments are specifically about online test 

multiple attempts and the reasons why some students did not meet the final course requirements (see discussion 

points) Absolute qualitative data enrich our understanding and improve AAS information for subsequent researchers 

to use for new studies. 

170

Summary 

This section summarizes the study’s results and discusses the weaknesses in the validity and reliability of the AAS, 

which emerge from the contextual and multidimensional characteristics of the evaluation in a non-experimental 

design. Then it discusses the importance of achieving a balance between the reliability and validity criteria in an 

Authentic Assessment System. 

The following section associates the weaknesses of the AAS to the Online Test modality, to the use of rubrics and 

self-evaluation and co-evaluation techniques, to the drop out rate, and to the time restrictions of an evaluation in 

context. The thoughts about these weaknesses give rise to suggestions for improving assessment systems in web 

based courses. 

Finally, suggestions are given for improving the validity and reliability of the AAS in a gradual process using time 

series design and maintaining the measurements in real contexts. Improvement was focused on the evaluation 

instruments and in the system’s design. The suggestions regarding the instruments were: to use integrated 

instruments, refined rubrics, and rigorous quality criteria. The suggestions for improving the design of the authentic 

assessment are: to reduce the number of specific objectives, to harmonize the individual component with the group 

component, if applicable, to include self-regulation procedures, to apply differentiated evaluation in accordance with 

the roles in the collaborative work and to the potentialities of the peer evaluators and/or experts, in order to favor a 

context of effectiveness and ecological validation. 

References 

Aiken, L. R. (1996). Tests Psicológicos y Evaluación, México. Prentice Hall. 

Bandura, A., & Walters, R.H. (1963). Social Learning and Personality Development, New York: Holt, Rinehart and 

Winston. 

Benzie, D. (1999). Formative evaluation: Can models help us to shape innovative programmes? Education and 

Information Technologies, 4 (3), 251-262. 

Chang, C.-C. (2002). Assessing and Analyzing the Effects of WBLP on Learning Processes and Achievements: 

Using the Electronic Portfolio for Authentic Assessment on University Students' Learning. Paper presented at the 

EdMedia 2002 Conference, June 24-29, 2002, Denver, CO, USA. 

Chong, S.M. (1998). Models of Asynchronous Computer Conferencing for Collaborative Learning in Large College 

Classes. In C. J. Bonk & K. S. King (Eds.), Electronic collaborators: learner-centered technologies for literacy, 

apprenticeship, and discourse. Mahwah, N.J.: Lawrence Erlbaum Associates, 157-182. 

Clarke, M., Butler, C., & Schmidt-Hansen, P. (2004). Quality assurance for distance learning: A case study at Brunel 

University. British Journal of Educational Technology, 35 (1), 5-11. 

Collis, B., De Boer, W., & Slotman, K. (2001). Feedback for web-based assignments. Journal of Computer Assisted 

Learning, 17 (3), 306-313. 

Condemarin, M., & Medina, A. (2000). Evaluación Auténtica de los Aprendizajes: Un medio para mejorar las 

competencias en lenguaje y comunicación, Santiago de Chile: Editorial Andrés Bello. 

Downs, E., Carlson, R. D., Repman, J., & Clark, K. (1999). Web-Based Instruction: Focus on Learning. Georgia 

Southern University. Paper presented at the SITE Conference, February 28-March 4, 1999, San Antonio, TX, USA. 

Duart J., & Sagrá, A. (2000). La formación en web: del mito al análisis de la realidad, retreived October 15, 2007, 

from http://cvc.cervantes.es/obref/formacion_virtual/campus_virtual/sangra2.htm. 

Fenwick, T., & Parsons, J. (1998). Starting with our stories: Towards more authentic assessment in adult education. 

Adult Learning, 26 (4), 25-30. 

171

Frederiksen, J.R., & Collins, A. (1989). A systems approach to educational testing. Educational Researcher, 18, 27- 

32. 

Gatlin, L., & Jacob, S. (2002). Standards-Based Digital Portfolios: A Component of Authentic Assessment for 

Preservice Teachers. Action in Teacher Education, 23 (4), 28-34. 

Gipps, C. (1994). Beyond Testing: Towards a Theory of Educational Assessment, London, The Falmer Press. 

Herman, J., Aschbacher, P., & Winters, L. (1997). A practical guide to alternative assessment, CA: CRESST. 

Herrington, J., & Herrington, A. (1998). How university students respond to a model of authentic assessment. Higher 

Education Research and Development, 17 (3), 305-322. 

Kroll, D., Masingila, O., & Mau, S. (1992). Grading Cooperative Problem Solving. The Mathematics Teacher, 85 (8) 

617-626. 

Lajoie, S. (1991). A framework for authentic assessment in mathematics. NCRMSE Research Review, 1 (1), 6-12. 

Lave, J. (1991). Situating Learning in Communities of Practice. In L. B. Resnick, J. M. Levine and S. D. Teasley 

(Eds.), Perspectives on socially shared cognition, Washington, DC: American Psychological Association, 63-82. 

Linn, R. L., Baker, E., & Dunbar, S. (1991). Complex, performance-based assessment: Expectations and validation 

criteria. Educational Researcher, 20 (8), 15-21. 

Meyer, C.A. (1992). What’s the difference between authentic and performance assessment? Educational Leadership. 

49 (8). 39-40. 

National Center for Fair and Open Testing (1992). What Is Authentic Assessment? Cambridge, MA. NCFOT. 

Newmann, F., & Wehlage, G. (1993). Five Standards of Authentic Instruction. Educational Leadership, 50 (7), 8-12. 

Olson, T., & Wisher, R. (2002). The Effectiveness of Web-Based Instruction: An Initial Inquiry. International 

Review of Research in Open and Distance Learning; 3 (2), retrieved October 15, 2007, from 

http://www.irrodl.org/index.php/irrodl/article/view/103/182. 

Orde, B. (2001). Online course development: summative reflections. International Journal of Instructional Media, 

28 (4), 397-403. 

Peterson, M. (2000). Electronic Delivery of Career Development University Courses. In J.W. Bloom, & G. R. Walz 

(Eds.), Cybercounseling and Cyberlearning: Strategies and Resources for the Millennium, Alexandria, VA: 

American Counseling Association, 143-159. 

Phipps, R., & Merisotis, J. (1999). What’s the difference? A review of contemporary research on the effectiveness of 

distance learning in higher education, Washington DC: The Institute for Higher Education Policy. 

Reeves, T. C., & Okey, J. (1996). Alternative assessment for constructivist learning environments. In B. Wilson, 

(Ed.), Constructivist learning environments, Englewood Cliffs, NJ: Educational Technology, 191-202. 

Resnick, L.B., & Resnick, D.P. (1992). Assessing the thinking curriculum: New tools for educational reform. In B.R. 

Gilford & M.C. O´Connor (Eds.), Changing assessment: Alternative views of aptitude, achievement and instruction, 

Boston: Kluwer, 37-75. 

Roschelle, J. (1992). What Should Collaborative Technology Be? A Perspective from Dewey and Situated Learning. 

ACM SIGCUE Outlook, 21 (3), 39-42. 

Ryan, W. J. (2001). Comparison of Student Performance and Attitude in a Lecture Class to Student Performance and 

Attitude in a Telecourse and a Web-Based Class, Ph.D. Dissertation No. ED467394, Nova Southeastern University. 

172

Salomon, G. (1992). What Does the Design of Effective CSCL Require and How Do We Study Its Effects? ACM 

SIGCUE Outlook, 21 (3), 62-68. 

Scardamalia, M., & Bereiter, C. (1996). Computer Support for Knowledge-Building Communities. In T. Koschmann 

(Ed.), CSCL, theory and practice of an emerging paradigm, Mahwah, NJ: Lawrence Erlbaum Associates, 249-268. 

Schurr, S. (1999). Authentic Assessment From A to Z, USA: National Middle School Association. 

Stake, R. (1995). The art of case research, Thousand Oaks, CA: Sage. 

Straetmans, G., Sluijsmans, D., Bolhuis, B., & Van Merrienboer, J. (2003). Integratie van instructie en assessment in 

competentiegericht onderwijs [Integration of instruction and assessment in competency-based education]. Tijdschrift 

voor Hoger Onderwijs, 21, 171-198. 

Torrance, H. (1995). Evaluating authentic assessment: Problems and possibilities in new approaches to assessment, 

Buckingham: Open University Press. 

Tucker, S. (2000). Assessing the Effectiveness of Distance Education versus Traditional On-Campus Education. 

Paper presented at the Annual Meeting of the AERA, April 24-27, 2000, New Orleans, Louisiana, USA. 

Vygotsky, L. S. (1985). Thought and Language, Cambridge, MA: MIT Press. 

Weiss, C. (1997). Investigación Evaluativa. Métodos para determinar la eficiencia de los programas de acción (4 th 

Ed.), Mexico: Editorial Trillas. 

White, S. (1999). The effectiveness of web-based instruction: A case study. Paper presented at the Annual Meeting 

of the Central States Communication Association, April, St. Louis, MO, USA. 

Wiggins, G. (1989). A True Test: Toward More Authentic and Equitable Assessment. Phi Delta Kappan, 70 (9), 

703-713. 

Wiggins, G. (1998). Educative Assessment. Designing Assessments to Inform and Improve Student Performance, San 

Francisco: Jossey-Bass. 

Wiggins, G. (1990). The case for authentic assessment. Practical Assessment, Research & Evaluation, 2 (2), 

retrieved October 15, 2007, from http://pareonline.net/getvn.asp?v=2&n=2. 

Wiggins, G. (1993). Assessment: Authenticity, context, and validity. Phi Delta Kappan, 75 (3), 200-208. 

Wilkerson, J., & Elkins, S. (2000). CAD/CAM at a Distance: Assessing the Effectiveness of Web-Based Instruction 

to Meet Workforce Development Needs. Paper presented at the Annual Forum of the Association for Institutional 

Research, May 21-24, 2000, Cincinnati, OH, USA. 

Woolf, B. P., & Regian, J. W. (2000). Knowledge-based training systems and the engineering of instruction. In S. 

Tobias and J. Fletcher (Eds.), Training and retraining: A handbook for business, industry, government, and the 

military, New York: Macmillan Reference, 339-356. 

Weller M. (2002). Assessment Issues on a Web-based Course. Assessment & Evaluation in Higher Education, 27 

(2), 109-116. 

Yin, R. (1984). Case study research: Design and methods (1 st Ed.), Beverly Hills, CA: Sage. 

Yin, R. (1994). Case study research: Design and methods (2 nd Ed.), Beverly Hills, CA: Sage. 

Young, M.F. (1995). Assessment of situated learning using computer environments. Journal of Science Education 

and Technology, 4 (19), 89-96. 

173

Bottino, R. M., & Robotti, E. (2007). Transforming classroom teaching & learning through technology: Analysis of a case study. 


Transforming classroom teaching & learning through technology: Analysis of a 

case study 

Rosa Maria Bottino and Elisabetta Robotti 

Consiglio Nazionale delle Ricerche, Istituto Tecnologie Didattiche, Genova, Italy // Tel. +39 01 06 47 56 76 // Fax. 

39 01 06 47 53 00 // bottino@itd.cnr.it // robotti@itd.cnr.it 

ABSTRACT 

The paper discusses the results of a research project based on the field testing of a course aimed at developing 

arithmetic problem solving skills in primary school pupils. The course was designed to incorporate e-learning 

techniques, including the use of ARI@ITALES authoring tools. These tools allowed the integration in the 

course of constructivist activities based on interaction with a set of different microworlds. The aims of the 

project were twofold: to analyse how the adopted approach and tools could help the teacher design and manage 

classroom activities integrating technology; and to evaluate the effectiveness of the ARI@ITALES tools for 

supporting pupils’ acquisition of mathematical skills. 

Keywords 

E-learning, Authoring software, Primary education, Numeracy, Problem Solving 

Introduction 

Despite the positive results obtained in a number of experimental settings and the large investments made by many 

governments for equipping schools with hardware and software, it can be said that the integration of computer 

technologies in the classroom still remains a limited phenomenon at primary school level (see, for example, Venezky 

& Davis, 2002; Sutherland, 2004). This is true also for disciplines like mathematics, which, from the beginning, has 

been one of the school subjects that has attracted considerable educational research attention concerning the 

development and use of ICT tools (Artigue, 2000; Lagrange, Artigue, Laborde & Trouche, 2001). One of the main 

reasons for this low impact is that technology has often been introduced as an addition to an existing, unchanged 

classroom setting (De Corte, 1996). Even though it now appears necessary to adopt a more integrated vision in which 

ICT is considered in conjunction with educational strategies, contents and activities (Bottino, 2004), this change of 

perspective is unlikely to be straightforward and means taking different perspectives into account. The perspective 

considered in this paper is that of teachers and of the difficulties they encounter in integrating ICT tools in their 

classroom practice. In order to bring meaningful innovation to teaching and learning processes, changes are required 

in content, organization and management of classroom activity – changes that cannot be accomplished effectively by 

a teacher operating alone. 

In this paper the use of an e-learning approach, supported by specifically designed authoring tools, it is examined 

with the objective to understand how effectively it can support the teacher in the development of classroom activities 

integrating technology. In particular, the paper analyses the design, implementation and field testing of an online 

course aimed to promote the development of arithmetic problem solving skills in primary school pupils. This course 

was built using a set of authoring tools (ARI@ITALES tools) that were developed for implementing interactive 

constructivist activities in mathematics. 

In the following, the ARI@ITALES tools are briefly examined as well as the approach used to build the course using 

these tools. Selected findings from the field testing of the course with a fourth-grade primary school class (age 9-10) 

are then analysed. This analysis focuses on understanding whether the approach followed in designing the course and 

the authoring tools adopted effectively supported teachers’ efforts in integrating new technologies in their classes. 

Furthermore, the paper considers whether this integration had a positive impact on students’ learning of mathematics 

content. 

Background 

Our research group has been involved in the design and experimentation of mathematics educational software since 

several years. In particular, the ARI-LAB system has been developed to promote arithmetic problem solving skills in 

pupils of primary and lower secondary school. 






174

ARI-LAB is a multi-environment open system based on constructive principles that was widely experimented and 

used in a variety of class situations (Bottino & Chiappini, 2002). Such experiments showed its remarkable cognitive 

potentialities but evidenced also that its effective use in classroom practice (like the use of any other open interactive 

environment) requires the teacher a considerable work for integrating it in class activities that effectively impact on 

students’ learning. The design and the management of such activities are often difficult to be performed by a single 

teacher without a proper support. 

Such considerations led us to design and to implement tools and methodologies able to sustain the teachers in 

carrying out ICT-based activities in their classes. We performed this work within ITALES, a European Commission 

co-funded project (IST-2000-26356) with several objectives. Among them, the development of a set of authoring 

tools to enable teachers to prepare digital content for personalised learning, and the development of a new learning 

management system that can be used by teachers to plan and build their own online courses. 

Within ITALES, our research group designed and implemented ARI@ITALES tools based on the ARI-LAB2 

system, the new current version of the ARI-LAB system. Information on ARI-LAB2 system and on ARI@ITALES 

tools can be found at: http://www.itd.cnr.it/arilab/. 

ARI@ITALES is a set of authoring tools for building and implementing e-learning activities in mathematics. Using 

ARI@ITALES tools, e-learning courses have been designed and tested in real class situations. One such course is 

analysed in this paper. 

It is worth noting that the term e-learning is not used here to indicate a particular delivery strategy but rather an 

evolution in educational organization aimed at supporting teaching and learning processes through the use of social 

and technological resources (Alvino & Sarti, 2004). ARI@ITALES tools allow an innovative shift towards elearning 

since they permit to design courses which integrate a constructivist learning approach (activities in 

microworlds) with the learning object philosophy (production of digital resources for online learning). 

Field experiments involving the use of ARI-LAB and ARI@ITALES tools have been carried out not only at the local 

level but also considering a wider perspective. For example, within the TELMA (Technology Enhanced Learning in 

MAthematics) joint research activity of the Network of Excellence Kaleidoscope (http://www.noe-kaleidoscope.org - 

accessed January 2007) a cross-experimentation project was organised to compare the different approaches to the 

study of ICT-based teaching and learning environments adopted by the European teams participating to TELMA. 

The key idea was that each team designed and implemented a teaching experiment in a real classroom making use of 

an ICT-based tool developed by another team. ARI-LAB has been, in particular, experimented by French and Greek 

teams. First results of such cross-experimentation project are reported by Artigue et al. (2006; 2007). One of the 

findings that has been pointed out is that it is necessary to support teachers not only with knowledge about how to 

use a software tool but also with its key theoretical and educational principles, explicating the settings and the 

pedagogical practices which can foster its educational potential. This suggests that the tools and the approach here 

reported can be useful for supporting teachers in designing not only specific learning activities integrating 

technology but also in building the whole pedagogical itineraries where such activities are to be inserted in order to 

suit stated learning objectives. 

The ARI@ITALES tools 

ARI@ITALES is a set of authoring tools for building and implementing mathematics e-learning activities at primary 

and lower secondary school levels (from grades 2 to 8). 

The ARI@ITALES tools were designed to support the teacher in the preparation of mathematics learning activities 

based on interaction with a number of microworlds. In these activities students can approach abstract and formal 

concepts (e.g. the number concept) by working with concrete representations (e.g. money, calendar, abacus, etc.). 

The tools of ARI@ITALES are: the Text Editor, the Microworlds, the Solution Sheet, the Simulator, and the 

Player. These are currently available in three languages: Italian, English and Spanish. 

175

The Text Editor allows the editing and saving of problem texts as objects to be included in a course. When a text is 

saved, the Text Editor automatically inserts a command that allows the course participant to solve the problem by 

accessing the Solution Sheet and then the Microworlds. The Solution Sheet and Microworlds are automatically 

instantiated with the text of the problem at hand. 

Microworlds are mediating tools for building the solution to a problem. Within microworlds, the user, through the 

creation and manipulation of computational objects, can visually represent problem situations in a variety of concrete 

contexts, which are meaningful also from the mathematics point of view. The Microworlds currently available in 

ARI@ITALES are: “Euro”, “Calendar”, “Abacus”, “Number Building”, “Number Line”, “Graphs”, “Spreadsheet”, 

“Arithmetic Operations”, “Fractions”, and “Arithmetic Manipulator”. 

Some microworlds (such as Euro or Calendar) were designed to model common everyday situations such as ‘buying 

and selling’ or ‘time’ problems. For example, to solve a problem involving counting days, students can enter the 

Calendar microworld and visualise a month, mark intervals of days, pass from one month to another, etc. Similarly, 

to solve a problem involving a money transaction students can enter the Euro microworld and generate Euros to 

represent a given amount, move coins on the screen, change them with other Euro coins or banknotes of an 

equivalent value, etc. Figure 1 shows the interface of the Euro microworld with an example of interaction. 

Figure 1. Interface of the Euro Microworld with an example of interaction 

Others microworlds are designed to offer different ways of representing and manipulating numbers (Number Line, 

Abacus, Graphs), of performing calculations (Spreadsheets, Arithmetic Operations), and of dealing with more 

abstract mathematical concepts (Fractions, Arithmetic Manipulator). During problem solving, microworlds allow the 

users to manipulate computational objects and interact with them using operational tools specific to each microworld. 

While interacting with microworlds, users receive various kinds of feedback that may foster the emergence of 

goals for problem solution and the construction of meaning for the strategies developed. For example, in the Euro or 

in the Abacus microworld, it is possible to select a coin or group of coins (or a configuration of balls in the abacus) 

and to hear the corresponding amount pronounced orally by means of a voice synthesizer incorporated in the system. 

Moreover, in each microworld, some tasks performed by the users are controlled by a set of rules integrated in the 

system that prevent them from taking specific incorrect steps. For example, if the users try to perform an incorrect 

change of coins, or of balls in the Abacus, the system prevents them to continue and addresses a specific error 

message. 

176

In the Solution Sheet it is possible to elaborate the solution process enacted within the microworlds, transforming it 

into a product to reflect on and to share with others. The underlined metaphor is that of the math workbooks where 

usually students do exercises and build solutions to problems. In the solution sheet users builds up a solution to the 

problem at hand by copying into this space the visual representations produced in the microworlds that they consider 

meaningful for working towards the solution. Students can employ verbal language and arithmetic symbolism to 

comment on the graphical representations copied and thus to explain their solution. This can be done by means of the 

“post-it” function. This function allows editing a short note, a comment, or, also, a mathematical expression, that it is 

possible to add, remove, correct, and move about in the solution space. Figure 2 shows the interface of the solution 

sheet with a solution produced by a student tackling a “purchase and sale” problem. 

Figure 2. Interface of the Solution Sheet with an example solution 

The Simulator allows the building of a ‘simulation’, in that it automatically records a sequence of actions performed 

by the users (in one or more microworlds and/or in the Solution Sheet). These can then be viewed as a sort of movie 

in the Player environment, possibly with an accompanying audio commentary. A simulation can be used to illustrate 

a concept, to describe how a specific problem can be solved, or, also, to explain the functioning of a microworld. 

The course on arithmetic problem solving 

This course was designed to introduce 9-10 year-old students to the solution of arithmetic problems of additive and 

multiplicative structure. The course was designed to be carried out in the classroom under teacher supervision, not 

for individual distance learning outside this context. 

Objectives 

The main research objectives underlying the design of the course on arithmetic problem solving were twofold: firstly 

to study whether and to what extent ARI@ITALES tools could support teachers in designing and managing 

classroom activities integrating technology and to assist them in overcoming some of the difficulties they usually 

encounter in this process; and secondly, to understand whether the designed activities could enhance students’ 

learning of arithmetic concepts and problem solving strategies. 

177

Mathematical content and structure 

The course consisted of three modules comprising activities built with the ARI@ITALES tools: explanations, 

simulations, examples, problems to be solved, solutions to be completed by pupils, tests, consolidation exercises. All 

the modules were designed in collaboration with the two teachers who then ran the course in their class. 

Simulations were prepared that explained how to use a microworld or a specific function therein (e.g. the change 

function in the Euro microworld) as well as to introduce concepts like monetary equivalence. Examples of solved 

problems were included to present a new solution strategy or to introduce a new mathematics concept. Tests were 

developed to assess the learning of specific concepts and procedures and to provide diagnostic feedback, with 

suggestions for further activities. Problems (mainly verbal problems) were formulated according to questions of 

increasing difficulty. Sometimes different versions of the same problem were proposed to different groups of 

students according to their level of ability. The course was initially assembled using the LRN Editor of Microsoft 

LRN Toolkit 3.0, then using the ITALES course assembly tool. 

As to the mathematics learning objectives, the course was mainly focused to the development of arithmetic abilities 

of increasing level of formalization: counting, reading and writing numbers and making calculations; building 

strategies in microworlds for solving concrete arithmetic problems; converting the solution produced in a microworld 

into written verbal language and then into arithmetical relations; recognizing invariant properties in the structure of a 

problem and in its solution (e.g. reduction to the unit for proportionality problems). 

More specifically, the activities proposed in Module 1 were mainly dedicated to develop counting capabilities and to 

solve additive and multiplicative purchase and sales problems by applying different strategies (e.g. 

total/part/remainder, completion, containment, partition). These activities entailed the use of the Euro and Abacus 

microworlds. Module 2 activities were aimed at reinforcing counting strategies in various situations (e.g. counting 

days and intervals of days) and at introducing concepts such as multiple, sub-multiple, and lowest common 

denominator. They mainly involved the use of the Calendar microworld. Module 3 was designed to introduce 

concepts pertaining to data representation by means of tables, histograms and graphs and involved the use of the 

Graph microworld. The calculation of parameters such as mode, arithmetic mean and median were introduced using 

the Spreadsheet microworld. Exercises were also included to foster reflection on simple formulas involving variables 

so as to make pupils aware of possible generalizations (e.g. direct and inverse proportion problems). 

The field testing 

Methodology 

The course was tested in 2004 in a fourth-grade class composed of 20 pupils. The teaching experiment took place in 

the school computer laboratory equipped with 12 computers with web connections. For the lab sessions, the students 

were divided into two groups: in this way each pupil had a computer at his/her disposal. Each group attended the lab 

two hours per week for four months. 

The experiment took place during normal school hours and was carried out by the two class teachers, alternatively 

one taking the group in the lab while the other followed the other group in class. Elisabetta Robotti participated in all 

the laboratory sessions as observing researcher. During class work, when all pupils were present, the teachers always 

sum up the activities performed in the laboratory and discussed them with the whole class. 

By the end of the field experiment all the pupils had completed the first two modules of the course; time limitations 

prevented all but a few from completing Module 3. Consequently in the following section data from this last Module 

are not presented. 

Data gathering 

In order to collect data for evaluating the teaching experiment, two different types of observational assessment sheet 

were prepared. Such sheets were filled in for each pupil and for each laboratory session. 

178

The first type of sheet (‘course sheet’) was designed to measure the efficacy of the course organization and 

components (simulations, problems, explanations, tests, etc.) as a support for the teacher. The second type of sheet 

(‘tools sheets’) was designed to measure the efficacy of the ARI@ITALES tools as a support for students in carrying 

out arithmetic problem solving activities. 

In both types of sheet, the observations made during the experiment were clustered around three different issues: 

‘ease of use’, ‘impact’, and ‘effectiveness’. Each aspect was attributed, by the observing researcher, a numerical 

score from 1 (poor) to 4 (very good); space was also provided for comments. Issues considered in the sheets are 

mediated from Technology Acceptance Model theory (Davis, 1989). Such theory proposes a model to describe how 

users come to accept and use a technology, and states that acceptance and use of a technology was determined by 

two factors: perceived usefulness and perceived ease of use. In the analysis here reported the meaning of these 

crucial factors was adapted taking into account the specific educational and pedagogical objectives of the study 

under examination. More specifically, in the course sheets, ‘ease of use’ indicated the degree of difficulty 

encountered by a pupil in dealing with an activity in the course, e.g. downloading a simulation on the computer and 

using it. ‘Impact’ gave a general evaluation of the pupils’ reaction to a course component, especially when they used 

it for the first time. For example, the impact of a simulation was evaluated in terms of the level of acceptance 

(whether the pupils liked it), possible impeding factors like length (overly long simulations might be boring, overly 

short ones vague), and the clarity of the explanation. ‘Effectiveness’ estimated whether a component fulfilled the aim 

for which it was designed in a given context. For example, simulation effectiveness was measured in terms of its 

success in explaining a concept and in helping the pupil understand errors. 

The Tools Sheets evaluated how ARI@ITALES tools supported students in the problem solving process. Particular 

attention was dedicated to the main functions of the Solution Sheet and of the Microworlds. This evaluation was 

aimed at analysing the characteristics of the computer transposition of arithmetic knowledge and at studying whether 

the design of the learning activities successfully exploited those characteristics. 

In the Tools sheets, ‘ease of use’ and ‘impact’ indicated respectively the degree of ease with which a function was 

used and the way in which pupils perceived it. ‘Effectiveness’ measured the extent to which the functions 

incorporated in the Solution Sheet and in the Microworlds were useful in promoting the development of specific 

arithmetic competencies and in helping students validating their actions. For example, it was demonstrated that some 

functions in the Euro microworld, like generating and moving coins on the screen, fostered the development of 

counting strategies (e.g. making groups of the same value, completing an amount to obtain a whole, etc.). Similarly 

functions in the Solution Sheet like the post-it function promoted pupils’ abilities in explaining and verbalizing their 

solution strategies. 

Tables 1 and 2 show an elaboration of the evaluation data collected with the course and tools sheets. Table 1 shows 

the evaluation given to the considered issues at three different moments in the experiment: the beginning and the 

conclusion of Module 1 and the end of Module 2. Table 2 shows the development of the pupils’ arithmetic 

competencies at the beginning and at the end of Module 1. 

Course 

components 

Simulations 

Tests 

Table 1. Results concerning the course components 

Ease of use Impact Effectiveness 

Module1 Module1 Module2 Module1 Module1 Module2 Module1 Module1 Module2 

Start End End Start End End Start End End 

(2) (2) 17% (2) 22% (2) (2) 17% (3) 100% (3) 55% (3) 70% For first 

100% (3) 82% (4) 77% 100% (3) 82% 

(4) 30% (4) 29% explanations: 

(2) 11% 

(3) 33%; 

(4) 55%; 

For help: 

(0) 45% 

(1) 55%; 

(2) 25% (2) 3% 

(3) 50% (3) 17% 

(3) 32% (3) 35% 

(3) 40% (3) 8% 

(4) 35% (4) 82% 

(4) 23% (4) 44% 

(4) 30% (4) 88% 

179

Consolidation exercises 

Solution completion exercises 

Explanations 

(2) 10% 

(3) 20% 

(4) 60% 


(3) 40% 

(4) 45% 


(3) 31% 

(4) 58% 

(2) 15% 

(3) 26% 

(4) 58% 

(2) 11% 

(3) 11% 

(4) 55% 

(3) 50% 

(4) 25% 

(2) 25% 

(3) 55% 

(4) 20% 

Used 

them: 

20% 

Did not 

use 

them: 

80% 

(3) 58% 

(4) 41% 

(2) 1% 

(3) 82% 

(4) 17% 

Used 

them: 

5% 

Did not 

use 

them: 

94% 

(2) 11% 

(3) 22% 

(4) 44% 

(3) 55% 

(4) 44% 

(4) 85% (4) 58% Intersection 

of time 

intervals 

(2) 11% 

(3) 11%; 

(4) 66%; 

Number 

composition 

and 

factorisation 

(1) 33% 

(2) 22%; 

(2) 25% 

(3) 70% 

(4) 5% 

(3) 50% 

(4) 50% 

Table 2. Results concerning ARI@ITALES tools 

ARI@ITALES Ease of use Impact Effectiveness 

Tools Module1 Module1 Module1 Module1 Module1 

Start End Start End Start 

Cut & Paste (2) 5% (4) 100% (2)5% (4) 100% (2)10% 

(3) 20% 

(3) 25% 

(3) 20% 

(4) 55% 

(4) 60% 

(4) 100% 

Solution Sheet 

Post-it (4) 65% (4) 100% (4) 65% (4) 100% Assigned Task 

(4) 45% 

Free use 

Accessing 

microworlds 

(2)10% 

(3) 25% 

(4) 50% 

(4) 100% (2)10% 

(3) 25% 

(4) 50% 


(4) 35% 

(2) 1% 

(3) 52% 

(4) 47% 

(3) 17% 

(4) 82% 

Module1 

End 


Free use 


(4) 33%; 

Partition of 

time 

intervals: 

(1) 11% 

(2) 22%; 

(3) 22%; 

(4) 33%; 

Numerical 

partition: 

(1) 33% 

(2) 11%; 

(4) 33%; 

Support for 

the learning 

of a new 

concept: 

(2) 55% 

(3) 11%; 

(4) 33%; 

Recalling of 

specific 

concepts: 

(3) 55% 

(4) 44%; 

180

Euro Microworld 

Abacus 

Microworld 

Voice 

synthesizer 

Moving 

coins on the 

screen 

Changing 

coins 

Changing 

balls in the 

abacus 

(4) 65% (4) 100% (4) 65% (4) 95% For validating: 

75% 

(2) 5% 


(4) 60% 

Module 2 

Start 

(2) 40% 

(3) 60% 

Discussion of findings 

For counting: 25% 

(4) 100% (4) 65% (4) 98% Arranging coins in 

groups: 0% 

Counting coins of 

the same value: 

75% 

Forming groups of 

integer value: 25% 

Counting in 

sequence: 55% 

For validating: 29% 

For counting: 64% 

Arranging coins in 

groups: 17% 

Counting coins of 

the same value: 21% 

Forming groups of 

integer value: 57% 

Counting in 

sequence: 15% 

Completion strategy: 

4% 

Total/part/reminder 

strategy: 11% 

Additive strategy: 

90% 

(4) 76% (4) 76% Understanding monetary equivalence: 

82% 

Changing coins in a suitable way for 

enacting a solution strategy: 47% 

division as repetition of subtractions: 41% 

Module 2 

End 

(3) 30% 

(4) 70% 

Module 2 

Start 

(2) 40% 

(3) 60% 

Module 2 

End 

(3) 70% 

(4) 30% 

Module 2 

Start 

Representing a 

number: 90% 

Performing 

additions and 

subtractions: 40% 

Module 2 

End 

Representing a 

number: 90% 

Performing additions 

and subtractions: 

80% 

In the following, a brief analysis, both qualitative and quantitative, of some findings from the experiment is provided 

linking them with the objectives of the project. This analysis provides indications on the course and on its 

components that may give suggestions for future work in the field. 

Supporting teachers in the design and management of classroom activity 

First of all, the teachers appreciated the opportunity that the course offered for better following the learning rhythm 

of each pupil and for monitoring pupils who had been absent from time to time. Although the students participated in 

class activities, the course allowed them to follow the proposed learning itinerary on an individual basis. In this way 

students who missed classes could resume their learning itinerary without gaps, and those who experienced 

difficulties with some concepts could go back to pertinent explanations, simulations and examples. 

The teachers also appreciated the possibility of designing activities to meet the needs of different students. For 

example, they found it useful to submit different versions of the same problem to pupils with different ability levels. 

Moreover, they considered it very useful to be able to save on their computers the work completed by each student 

during a session. In this way they could check and compare the different solutions and thus personalize the activities 

better during subsequent lessons. 

181

As to course components, the activities entailing completion of a partially implemented solution proved to be 

effective for gradually consolidating pupils’ acquisition of mathematical concepts. These activities were 

autonomously and correctly completed by 75% of the pupils in the initial sessions of the course and by 99% of them 

in the final sessions. Moreover, while at the beginning of Module 1 only 5% of pupils obtained a very good 

evaluation (score 4), at the end of this module this percentage rose to 47%. 

Simulations proved to be quite useful for helping the teacher introduce a concept or a particular function of a 

microworld. Table 1 shows that, at the beginning of Module 1, simulations were perceived as effective by 30% of 

pupils. At the end of Module 2, this percentage increased to 55%. Nevertheless a significant percentage of pupils 

(45%) did not use simulations autonomously, after a first launch attempt, to recall a concept or a procedure, 

preferring to ask their classmates or the teacher instead. This was probably due to the fact that launching and 

watching a simulation was not straightforward and so many pupils avoided repeating this process. 

The tests helped to verify the acquisition of specific procedures or concepts (e.g. ‘changing’ coins in the Euro 

microworld or balls in the Abacus). The tests were designed not only to provide evaluation feedback but also to 

introduce the pupil to specific remedial activities. This approach proved effective for the management of the activity, 

since the teacher did not have to rush from one pupil to another for checking and offering suggestions. The data show 

that, at the beginning of Module 1, 55% of pupils completed the tests correctly. At the end of this Module this 

percentage rose to 79%. 

The ARI@ITALES microworlds have specific functions for validating crucial arithmetic competencies. For 

example, as said before, in the Euro microworld it is possible to select a coin or group of coins and to hear the 

corresponding amount pronounced by means of a voice synthesizer incorporated in the system. During the 

experiment, this function took on a crucial role in helping the pupil to learn how to count money. The voice 

synthesizer allowed pupils to compare autonomously what they thought they had done (e.g. generating 1,50 euro 

worth of coins) with what they had actually done (generating a sum of 1,05). This allowed pupils to correct their 

work autonomously by practising the rules governing the counting of coins. At the same time, the teacher was freed 

from the need to check the students’ work for errors. The voice synthesizer was mainly used at the beginning of the 

course, when pupils had to strengthen counting and representation skills. As work progressed, they resorted to it less 

and less, although some pupils continued to use it as part of a trial and error strategy so as to avoid the effort of 

counting. This brought to light the need to provide the teachers with a function for disabling the voice synthesis as 

they saw fit. 

During the course, the voice synthesis was also used for reading aloud problem texts. This proved useful for pupils 

who had difficulties due to reading or sight problems and who would otherwise have asked the teacher repeatedly to 

read the problem text aloud during the solution activity. So in this case the voice synthesizer made it easier for the 

teacher to manage class activities. 

Other feedback functions incorporated in ARI@ITALES microworlds also proved to be effective. For example, the 

feedback provided by the system when students perform a change of coins in the in Euro microworld or of balls in 

the Abacus. Since this feedback is formulated to help students identify the error committed, if any, and correct it, it 

was frequently exploited during the experiment. For example, the teachers designed and proposed additional practice 

exercises for students with difficulties and asked them to perform these autonomously, relying on the system’s 

feedback for checking their answers. 

Development of arithmetic problem solving skills 

Arithmetic problem solving is a field in which primary school students often experience difficulties, as indicated by 

many research studies and reported by many teachers (see, for example: Mullis, Martin, Foy, 2005). These 

difficulties have strong repercussions on students' self-esteem and future mathematics performance. Even at primary 

school level, students frequently perceive ‘doing maths’ as the execution of repetitive exercises according to formal 

rules whose meaning they often do not understand, or master only at the syntactic level. For many pupils problem 

solving is limited to ‘guessing’ the right arithmetic operation and carrying out the written calculations, since the 

semantics they associate with arithmetic symbols is poor and frequently limited to what the result of a computation 

182

denotes. These considerations highlight the importance of developing new methodologies and tools to better support 

the development of meanings for arithmetic operators and for problem solving strategies. 

One of the objectives of the work here reported was to study how ARI@ITALES tools could be used to assist pupils 

in visually representing and solving arithmetic problems. Visual representation systems play a central role in 

mathematics education as a way of linking the symbolic approach to mathematics concepts to perceptive experience. 

ARI@ITALES microworlds were designed to offer pupils a variety of computational objects for representing and 

manipulating problem situations and resolutions steps. 

Consider, for example, the development of counting strategies, which are crucial for arithmetic problem solving 

(Adetula, 1996). The Euro microworld allowed pupils to represent amounts concretely by means of coins. At the 

beginning of the course, 75% of the pupils counted coins in sequence, according to their value, and often made errors 

when dealing with an increasing number of coins or with higher amounts. Becoming familiar with the Euro 

microworld and taking advantage of the movement and change opportunities it offers, the pupils gradually learnt 

strategies better suited to facilitate the counting process. For instance, by moving coins on the screen, they composed 

groups of coins whose amount corresponds to an integer value, e.g. 1 euro, first using coins of the same value (e.g. 

groups of five 20-cent coins) and then coins of different values. The acquisition of progressively structured counting 

strategies was assisted by inserting examples of problems solved in the course. Looking at these examples, pupils 

were exposed to new ways of arranging and counting coins and banknotes. These skills were then consolidated by 

proposing partially solved problems that pupils had to complete. At the end of Module 1, 57% of pupils were able to 

count mentally without moving coins on the screen since they had internalised the process of moving and grouping 

coins and only used the mouse pointer to support the counting process and the voice synthesizer to validate their 

results. 

As to the development of solution strategies, the possibility of moving coins on the screen helped the pupils to put in 

action total/part/remainder strategies and supported the attribution of a concrete meaning to the obtained groups of 

coins (the part, the remainder). 

The generation of coins facilitated pupils in mastering completion strategies in additive problems. These strategies, 

which are crucial for mental calculation, are usually rather difficult for pupils to control. With the support of the Euro 

microworld, they were able to control all the phases of the process. For example, starting from a given quantity, they 

first generated the cents necessary to reach the successive decimal and then the decimals needed to reach the whole, 

and so on until they reach the target amount. 

The change function of the Euro microworld supported the learning of the additive composition and decomposition 

of numbers by means of monetary equivalence. Results showed that a good percentage of pupils acquired these 

competencies from the early sessions of the course (82%). Almost half of the pupils also changed coins correctly 

with given constraints (e.g. changing a given amount using the least number of coins); this demonstrated that they 

had achieved a firm grasp of the manipulation of coins and numbers. The change function was used by a 

considerable number of pupils (47%) to perform the partition of a given amount into groups of given value. It is 

worth noting that the direct intervention of the teacher helped to avoid some possible misuse of the microworld 

functions (e.g. the opportunistic use of the change function or of the voice synthesizer). 

The activities in the Abacus microworld were designed to support both the exploration of the rules involved in the 

decimal positional writing of numbers and the acquisition of concepts involved in adding and subtracting decimal 

numbers (e.g. the carry over concept). At the beginning of Module 2, about 90% of pupils represented correctly in 

the Abacus a monetary value previously built up with coins, and 40% were able to perform operations (additions and 

subtractions) by changing balls properly. At the end of Module 2, the percentage of students performing correct 

operations in the Abacus reached 80%. 

The ARI@ITALES Solution Sheet supported the elaboration of the solution process enacted within the microworlds 

by allowing pupils to describe the strategy they had adopted both by means of the visual representations obtained in 

the microworlds and by means of verbal and symbolic language (using the ‘post-it’ function). The possibility of 

comparing different representations for the same value (for instance copying in the Solution Sheet the representations 

obtained in different microworlds, e.g. the Euro, the Abacus, and the Number Line microworlds) was used in the 

course to induce pupils to think over the symbolic representation of decimal numbers and to develop the capacity to 

183

handle more formal representations and their related properties. At the end of the course, almost 90% of the pupils 

were able to shift correctly from the verbal representation of a number to its decimal writing and to describe verbally 

in the Solution Sheet a procedure previously accomplished in a microworld; they subsequently managed to formalize 

this description using arithmetic symbols. 

The activities in the Calendar microworld proved effective for developing meanings for some mathematical 

concepts, such as those of multiple, sub-multiple, and lowest common denominator (seen, for example, as the 

multiple interval of two or more time intervals). Then problems concerning the integer partition of a given amount 

were proposed. Eventually, more formal ways to tackle the concept were gradually introduced. Multiple problems 

were correctly solved by 80% of pupils, while 55% managed to solve problems related to the concept of lowest 

common denominator and integer partition. 

Conclusions 

Many research studies reveal that it is pointless from a pedagogical point of view to make computers and educational 

digital media available in schools if their use is not properly embedded in suitably articulated educational itineraries 

in which the whole learning context is taken into account, including the pedagogical and curriculum objectives, the 

tools and the way in which they are used, the teaching/learning paths, the different actors and their social 

relationships, etc. (Dias De Figueiredo & Afonso, 2006). Proper contextualization therefore becomes decisive in 

making educational software effective; otherwise, the potential of even the best program will remain largely 

unexploited. The design of effective contexts of use for ICT-based tools is a complex process that also requires 

changes in content, organization and management of classroom activity, innovations that are difficult for a teacher to 

accomplish effectively. One of the objectives of the project discussed here was to analyse whether an e-learning 

approach, supported by specifically designed authoring tools, can help the teacher in facing such changes. 

The analysis of the teaching-learning activity carried out during the class experiment pointed out elements of the 

course and of the ARI@ITALES tools that had proved to be effective in supporting the management of classroom 

activities and the development of students’ cognitive processes in arithmetic problem solving. 

For example, the experiment highlighted the crucial supporting role of the feedback provided by ARI@ITALES 

tools. Direct diagnostic feedback proved useful in the acquisition of specific skills or procedures and in preventing 

students from making further incorrect steps. Indirect diagnostic feedback, such as that provided by the voice 

synthesizer, was helpful for supporting pupils in validating their work, thus fostering the development of crucial 

competencies such as counting. Moreover, some course components, such as tests, were designed to provide 

feedback in a way that would help students to understand errors made and to guide them in correction. 

Backtracking and the possibility of revising the work previously done proved useful for supporting the ability of 

verbalizing and explaining the actions performed. This ability was also supported by the Solution Sheet, which 

allowed elaboration of the solution process enacted within the microworlds, transforming it into a product to reflect 

on and to share with others. 

Some of the activities proposed stimulated students’ attitude to anticipate mentally hypotheses and problem 

solutions. For example, exercises that involved completing a partially implemented solution, letting students 

concentrate on single steps, facilitated acquisition of gradually articulated solution strategies. 

A decisive role was played by the pupils’ interaction: they often discussed and exchanged opinions and advice on 

strategies to be used, and compared results. This important aspect of the activity was only partially supported by the 

technology used, since at the time the experiment was conducted the ITALES platform was not fully implemented 

and the online communication function could not yet be used. This was certainly a limit that prevented the design of 

course activities that exploit computer-mediated communication to promote mathematics learning. In previous work, 

the rich learning opportunities offered by such activities were analysed on the basis of a small-scale experiment 

carried out using a simple local network connection and an early version of the ARI-LAB system (Bottino, 2000). In 

future, a rethink of course design will be needed in order to include computer-mediated communication and 

collaboration activities. It can be said that the flexibility of the approach followed in course design will make it 

easier, than in a traditional setting, to modify and reorganise learning activities and to change their content. 

184

Finally, it is worth noting that, even though this paper mainly focuses on analysing aspects related to teachers’ 

management of ICT-based activities and on evaluating students’ learning of arithmetic concepts, future work in the 

field will examine how the adopted approach can mediate the growth of communities of researchers and teachers 

who collaboratively develop, share and discuss the design of classroom activities based on the use of technology. 

This analysis will be carried out under the REMATH European project (EC-IST-4-26751-STP) that it is currently 

under development. 


Special thanks to Roberto Carpaneto and Anna Macchello, teachers at the M. Mazzini Primary School in Genova. 

Their valuable collaboration and support was crucial in the design and testing of the course under examination. 

References 

Adetula, L. (1996). Effects of counting and thinking strategies in teaching addition and subtraction problems. 

Educational Research, 38, 183-198. 

Alvino, S., & Sarti, L. (2004). Learning Objects e Costruttivismo. In Consorzio Omniacom (Eds.), Atti Didamatica, 

Ferrara, Italy, 761-772. 

Artigue, M. (2000). Instrumentation Issues and the Integration of Computer Technologies into Secondary 

Mathematics Teaching, Developments in Mathematics Education in German-speaking Countries: Selected Papers 

from the Annual Conference on Didactics of Mathematics, retrieved October 15, 2007, from 

http://webdoc.sub.gwdg.de/ebook/e/gdm/2000/artigue_2000.pdf. 

Artigue, M., Bottino. R.M., Cerulli. M., Mariotti. M.A., & Morgan, C. (2006). Developing a joint methodology for 

comparing the influence of different theoretical frameworks in technology enhanced learning in mathematics: the 

TELMA approach. 17th ICMI Study: Technology Revisited, retrieved October 15, 2007, from 

http://www.math.msu.edu/~nathsinc/ICMI/. 

Artigue, M., Bottino, R.M., Cerulli, M., Georget, J.P., Maffei, L., Maracci, M., Mariotti, M.A., Pedemonte, B., 

Robotti, E., & Trgalova J. (2007). Technology Enhanced Learning in Mathematics: the cross-experimentation 

approach adopted by the TELMA European Research Team. La Matematica e la sua Didattica, 21 (1), 67-74. 

Bottino, R.M. (2000). Computer-based communication in the classroom: defining a social context. In Watson, D.M. 

& Downes, T. (Eds.), Communications and Networking in Education: Learning in a Networked Society, Dordrecht, 

The Netherlands: Kluwer Academic Publishers, 343-354. 

Bottino, R.M. (2004). The evolution of ICT-based learning environments which perspectives for the school of the 

future. British Journal of Educational Technologies, 35 (5), 553-567. 

Bottino, R.M., & Chiappini, G. (2002). Technological Advances and Learning Environments. In English, L. (Ed.), 

Handbook of International Research in mathematics Education, Mahwah, NJ: Lawrence Erlbaum, 757-786. 

Davis, F.D. (1989). Perceived usefulness, perceived ease of use and user acceptance of information technology. MIS 

Quaterly, 13, 319-340. 

De Corte, E. (1996). Changing views of computer supported learning environments for the acquisition of knowledge 

and thinking skills. In Vosniadou, S., De Corte, E., Glaser, R. & Mandl, H. (Eds.), International perspectives on the 

designing of technology-supported learning environments, Mahwah, NJ: Lawrence Erlbaum, 129–145. 

Dias De Figueiredo, A., & Afonso, A.P. (2006). Managing learning in virtual settings: the role of context, Hershey, 

PA, USA: Information Science Publishing. 

185

Lagrange, J. B., Artigue, M., Laborde, C., & Trouche, L. (2001). Meta study on IC technologies in education. 

Towars multidimensional framework to tackle their integration into the teaching of mathematics. In Heuvel- 

Panhuizen, M. v. d. (Ed.), Proceedings of the 25th Conference of International Group for Psychology of 

Mathematics Education, Utrecht, Pays Bas: Freudenthal Institute, Utrecht University, 1, 111-122. 

Mullis, I.V.S., Martin, M.O., & Foy, P. (2005). Findings from a developmental project, Chestnut Hill, MA: TIMSS 

& PIRLS International Study Center, Boston College. 

Salomon, G. (1996). Computers as a Trigger for Change. In Vosniadou, S., De Corte, E., Glaser, R. & Mandl, H. 

(Eds.), International Perspectives on the Desing of Technology-Supported Learning Environments, Hillsdale, NJ: 

Lawrence Erlbaum, 363-377. 

Sutherland, R. (2004). Designs for learning: ICT and knowledge in the classroom, Computers & Education, 43, 5-16. 

Venezky, R. L., & Davis, C. (2002). Quo vademus? The transformation of schooling in a networked world, 

OECD/CERI, retrieved October 15, 2007, from http://www.oecd.org/dataoecd/48/20/2073054.pdf. 

186

Lu, H., Jia, L., Gong, S.H., & Clark, B. (2007). The Relationship of Kolb Learning Styles, Online Learning Behaviors and 

Learning Outcomes. Educational Technology & Society, 10 (4), 187-196. 

The Relationship of Kolb Learning Styles, Online Learning Behaviors and 

Learning Outcomes 

Hong Lu 

Department of educational technology, Shandong Normal University, China // Tel: +86-531-86188575 // 

luhong_1968@yahoo.com 

Lei Jia 

Department of English, Shandong Normal University, China // Tel: +86-531-86181218 // krowbat@gmail.com 

Shu-hong Gong 

Department of educational technology, Shandong Normal University, China // Tel: +86-531-86182530 // 

gongshuhong@21cn.com 

Bruce Clark 

Faculty of Education, University of Calgary, Canada // Tel: +1-403-220-7363 // bclark@ucalgary.ca 

ABSTRACT 

This study focused on the relationship between Kolb learning styles and the enduring time of online learning 

behaviors, the relationship between Kolb learning styles and learning outcomes and the relationship between 

learning outcomes and the enduring time of a variety of different online learning behaviors. Prior to the 

experiment, 104 students majoring in Educational Technology completed Kolb’s Learning Style Inventory 

(KLSI). Forty students were chosen to be subjects in an online learning experiment. Results indicated that there 

was a significant effect of Kolb learning style on the total reading time and total discussion time of the subjects. 

Although there was no significant effect between Kolb learning styles and learning outcomes, data from the 

experiment showed that the mean of learning outcomes of Convergers and Assimilators was higher than that of 

Divergers and Accommodators. There were two models of linear regression between learning outcomes and the 

enduring time of different online learning behaviors. Both of them were significant at the 0.001 level, and they 

accounted for 54.9% and 60.8% of the variance of the dependent respectively. The findings of this study were 

instrumental to instructors and moderators of online courses. First, instructors using online courses should 

seriously consider the diversity of learning styles when designing and developing online learning modules for 

different students. Second, they should provide a large number of electronic documents for students and give 

enough time to let them absorb knowledge by online reading. These could be effective methods to improve the 

quality of online courses. 

Keywords 

Kolb learning styles, Online learning behaviors, Learning outcomes 

Introduction 

Although online learning was growing rapidly, the effect of it was not yet satisfactory. For example, some students 

often complained that they could not find sufficient online learning resources to support their online courses, whereas 

other students were restricted by what they felt as a lack of opportunities to communicate with their instructors 

(Huang, 2003). Leigle & Janicki (2006) offered solutions to these problems, arguing that by customizing learning 

modules for differing student types, the learning outcome would be increased. Based upon this solution, the present 

study focused on the relationship of Kolb learning styles, online learning behaviors and learning outcomes. It was 

hoped that this study could help instructors understand the function of learning style in an online learning 

environment and thus develop corresponding online learning modules for different students. 

Kolb Learning Style Model 

The Kolb learning style model was based on Kolb’s experiential learning theory. In this model, Kolb defined 

learning style on a two-dimensional scale based on how a person perceived and processed information. How a person 

perceived information was classified as concrete experience or abstract conceptualization, and how a person 






187

processed information was classified as active experimentation or reflective observation (Simpson & Du, 2004). 

Accordingly, Kolb (1985) described the process of experiential learning as a four-stage cycle involving four adaptive 

learning modes: Concrete Experience (CE), Reflective Observation (RO), Abstract Conceptualization (AC), and 

Active Experimentation (AE). CE tended towards peer orientation and benefited most from discussion with fellow 

CE learners. AC tended to be oriented more towards symbols and learned best in authority-directed, impersonal 

learning situations, which emphasized theory and systematic analysis. AE tended to be an active, “doing” orientation 

to learning that relied heavily on experimentation and learned best while engaging in projects. RO relied heavily on 

careful observation in making judgments. Kolb (1985) also identified four learning style groups based on the four 

learning modes: Divergers favored CE and RO, Assimilators favored AC and RO, Convergers favored AC and AE, 

and Accommodators favored CE and AE. 

Kolb Learning Style Inventory 

There were many different learning style models. Many of them were derived from a common ancestry and 

measured similar dimensions (Brown, 2005). Accompanied by a vast collection of learning models, there was also a 

wealth of confusing assessment tools, amongst which, Kolb Learning Style Inventory (KLSI) remained one of the 

most influential and widely distributed instruments used to measure individual learning preference (Kayes, 2005). 

The original KLSI encountered serious attacks because of its low test-retest reliability and limited construct validity 

(John et al., 1991). In 1985, Kolb and his associates revised the KLSI to improve and refine its psychometric 

properties (Smith & Kolb, 1986). 

Some researchers had examined and found support for the revised KLSI. Veres et al. (1991) examined the revised 

KLSI and found increased stability. They argued that the revised KLSI might well be useful to researchers, educators 

and practitioners. Raschick, Maypole & Day’s (1998) research found the revised KLSI a useful tool for optimizing 

the relationship between supervisors and their students. The tool enabled both groups to view learning as a four-step 

process that involved experiencing, reflecting, conceptualizing and creatively experimenting. Based on these 

favorable results, KLSI became a well-accepted instrument for this experiment. 

Review of Literature 

Some recent studies analyzed the online learning behaviors between Kolb learning style groups. Simpson & Du 

(2004) explored the effect of Kolb learning styles on students’ online participation and self-reported enjoyment 

levels in distributed learning environments. Multiple regression analysis found that learning style had a significant 

impact on the students’ participation and enjoyment level. Fahy (2005) conducted a study of the relations between 

Kolb learning style and online communication behavior, and found that Convergers demonstrated their willingness to 

spend more time and energy on the network itself. Liegle & Janicki (2006) found that learners classified as 

“Explorers” (Active experimenters) tended to create their own path of learning (learner control), while subjects 

classified as “Observers” (Reflective observers) tended to follow the suggested path by clicking on the “Next” button 

(system control) in a web-based training program. However, there were studies that reached opposite conclusions. 

For example, in the environment of hypermedia learning, Reed et al. (2000) argued that Kolb learning styles had no 

effect on the number of linear steps, which determined whether the steps were the next logical, sequentially forward 

movement, and on the number of nonlinear steps, which determined whether the steps were branches or sidetracks. 

Why did inconsistent results occur in above-mentioned studies? The answer was that the lack of effect of Kolb 

learning styles could be due to that those studies involved the research variables, such as linear steps vs. nonlinear 

steps, that Kolb instrument did not seem to measure (Miller, 2005). According to the view, if research variables of 

the experiment could be measured by the KLSI, the effect of Kolb learning style would appear. To test it, this study 

chose for the online learning behaviors related with KLSI to be researching variables. 

Research investigating the learning outcome in an online or a hypermedia environment also reached confusing 

conclusions. For instance, Melara (1996) examined the effect of Kolb learning styles on learner performance within 

two different hypertext structures, and showed no significant difference in achievement for learners of different 

learning style using either hypertext structure. Davidson-shivers et al. (2002) investigated the effect of Kolb learning 

styles on undergraduate writing performance in a multimedia lesson. No statistically significant difference in writing 

performance among the learning styles was found. Howard and colleagues (2004) argued that, even though 

188

significant learning occurred, no significant difference in achievement was observed within any Kolb's 

classifications. Miller (2005) found no effect of Kolb learning styles on performance when using a computer-based 

instruction system to teach introductory probability and statistics. Some experiments, however, showed positive 

effects of Kolb learning styles on students’ performance. Oughton & Reed (2000) tested 21 graduate students 

enrolled in a graduate hypermedia education class. They were told to construct concept maps on the term of 

hypermedia. Findings indicated that Assimilators and Divergers were the most productive on their concept maps. 

Terrell (2002) indicated that, in a web-based learning environment, students whose learning styles belonged to 

Convergers and Assimilators were likely to succeed than students whose learning styles belonged to Divergers and 

Accommodators. These confusing conclusions might be produced by various factors, such as the topic of the course 

and how grades are given. Therefore, practitioners of online courses had to take these factors into consideration when 

they hoped to make use of relevant research conclusions. 

Most of the previous research focused on investigating either online learning behavior or the learning outcome 

between Kolb learning style groups. Little research dealt with the relationship between online learning behavior and 

learning outcome. The purpose of this study was to prove if there were differences in the online learning behavior 

between Kolb learning style groups, and if so, whether the differences would lead to differences in the learning 

outcome. 

Research Variables 

Kolb Learning Style Groups 

The KLSI could identify subjects’ preference for perceiving and processing information. Subjects responded to the 

12-item Kolb instrument and were categorized as Convergers, Divergers, Assimilators or Accommodators. 

Online Learning Behavior 

Corresponding with the four learning modes in the Kolb learning style model, four different online learning 

behaviors were identified as research variables in this study. The enduring time of these variables was measured 

when the subjects designed Flash animations in an online learning environment. These types of behavior involved 

online discussion preferred by CE, online reading of electronic documents preferred by AC, Flash animation 

designing preferred by AE, and online observation of onscreen activities of other subjects preferred by RO. 

Learning Outcome 

The subjects’ task was to design one animation using Flash software. The animation included ten different text 

effects. Each effect of the animation counted as one point for a total of ten points. The learning outcome was to be 

measured according to the amount of text effects completed by the subjects. 

Research Questions 

The authors of this experiment hypothesized that the subjects with different learning styles tended to choose different 

online learning behaviors, which would subsequently result in different learning outcomes of the subjects. 

The research questions guiding this experiment were as follows: (1) What was the relationship between learning 

styles and the enduring time of online learning behaviors? (2) What was the relationship between learning styles and 

learning outcomes? (3) What was the relationship between learning outcomes and the enduring time of different 

online learning behaviors? 

189

Method 

Participants 

The participants were third-year undergraduate students in the Department of Educational Technology at Shandong 

Normal University in China. 104 students took part in the test of KLSI, 40 of whom, demonstrating evident 

preferences for learning styles, were chosen as subjects, with ten subjects in each learning style category. Table 1 

showed gender distributions of learning style categories. 

Table 1. Gender distribution of Kolb categories 

Male Female 

Convergers 5 5 

Divergers 4 6 

Assimilators 5 5 

Accommodators 4 6 

These subjects had grasped some basic computer knowledge, such as the application of Internet, the use of 

communicating software, drawing software and word processing software. They also acquired basic knowledge of 

Flash when they were freshmen. 

Procedure 

The experiment was performed in the university computer laboratory. The subjects were divided into ten groups. 

Each group contains four subjects including one Converger, one Diverger, one Assimilator and one Accommodator. 

Then they were given 120 minutes to perform the designated task. During the 120 minutes, these four subjects 

worked individually and each one met with one experimenter, who observed and recorded their behaviors. Four 

graduate students who were familiar with the application of Flash were arranged to communicate with the subjects 

through the instant messaging software of QQ. A website encompassing an electronic document of how to design the 

animation using Flash was also provided. This electronic document was a detailed guide of how to design the 

animation in the task. 

Initially, the subjects were given 20 minutes to respond to the task. They were required to do so individually, without 

the help of online consultations, observations or references. After this pretest, they were given a 10-minute break. 

Each subject was then administered a posttest lasting 90 minutes, to respond to the task again. This time they could 

discuss with the graduate students through QQ, observe the designing process of other subjects (The subjects were 

authorized to access to the onscreen operations of others with their own computers.), read the electronic document on 

the Internet or design Flash animations by themselves. During the posttest, the experimenter observed and recorded 

the participants’ enduring time in online discussion with the graduate students (total discussion time), the enduring 

time observing the onscreen activities of others (total observation time), the enduring time actively reading the 

electronic document (total reading time) and the enduring time designing Flash animations (total designing time). 

Data analyses were performed by using, the Statistical Package for the Social Sciences, for Windows ([SPSS] ver. 

13.0). 

Results 

The relationship between learning styles and the enduring time of online learning behaviors 

The relationship between learning styles and the enduring time of online learning behaviors was analyzed in one-way 

ANOVA. The analysis found that learning styles had no significant effect on total observation time and total 

designing time. In fact, all subjects spent more than 45 minutes on designing. Only five subjects spent one or two 

minutes on observing the onscreen activities of others and the rest spent no time on the observation. However, 

190

subjects with different learning styles demonstrated significant differences in the categories of total discussion time 

and total reading time. 

Table 2. ANOVA for different learning styles and the enduring time of online learning behaviors 

Convergers Divergers Assimilators Accommodators 

M 

(min) 

SD 

M 

(min) 

SD 

M 

(min) 

SD 

M 

(min) 

SD 

F Prob. 

Total 

discussion 

time 

Total 

7.8 3.2931 14.1 4.4083 6.4 3.3400 12.6 3.2042 10.617 0.000 

observation 

time 

0.2 0.6325 0.1 0.3162 0.3 0.6749 0.1 0.3162 0.347 0.791 

Total reading 

time 

Total 

20.9 3.9847 15.9 3.3813 21.2 2.7809 14.1 3.6652 10.525 0.000 

designing 

time 

55.1 6.7569 53.8 4.8717 56.9 5.3427 57.7 5.9451 0.929 0.437 

Table 3. Scheffé post hoc comparison of total discussion time 


(Prob.) 

(Prob.) 

(Prob.) 

(Prob.) 

Convergers 0.005 0.859 0.045 

Divergers 0.005 0.000 0.832 

Assimilators 0.859 0.000 0.006 

Accommodators 0.045 0.832 0.006 

Table 4. Sceffé post hoc comparison of total reading time 


(Prob.) 

(Prob.) 

(Prob.) 

(Prob.) 

Convergers 0.027 0.998 0.001 

Divergers 0.027 0.017 0.722 

Assimilators 0.998 0.017 0.001 

Accommodators 0.001 0.727 0.001 

Table 3 and Table 4 (the Scheffé post hoc comparisons) showed that two significant effects appeared between the 

subjects who favored abstract conceptualization (Convergers and Assimilators) and those who favored concrete 

experience (Divergers and Accommodators). That is, on the one hand, subjects identified as Convergers and 

Assimilators spent more time on online reading than those identified as Divergers and Accommodators. On the other 

hand, subjects identified as Divergers and Accommodators spent more time on online discussing than those 

identified as Convergers and Assimilators. Furthermore, significant differences were not found between gender and 

total discussion time (F(1,38)=0.041, p=0.841), total observation time (F(1,38)=0.009, p=0.926), total reading time 

(F(1,38)=0.432, p=0.515) and total designing time (F(1,38)=0.009, p=0.925). 

The relationship between learning styles and learning outcomes 

The subjects had learned some introductory knowledge of Flash in their first year of university. Owing to scarce 

practice during the following two years, only seven subjects finished one text effect in the pretest. Among them were 

two Convergers, one Diverger, one Assimilator and three Accommodators. No subject completed the ten effects in 

the posttest. The task seemed challenging to all subjects. 

191

Table 5. Learning outcomes of different learning styles 


M SD M SD M SD M SD 

Pretest 0.2 0.422 0.1 0.316 0.1 0.316 0.3 0.483 

Posttest 5.3 2.163 4.4 1.776 4.9 2.183 4.8 2.658 

Learning outcomes 5.1 1.912 4.3 1.567 4.8 2.044 4.5 2.224 

To analyze the relationship between learning styles and learning outcomes, each subject was categorized as 

demonstrating either a high or low learning outcome. High learning outcome was defined as a learning outcome 

which was equal to or more than five points (higher than the mean learning outcome for subjects, M=4.68). The 

result of chi-square test showed that there was no significant association between learning styles and learning 

outcomes (χ 2 (3, N=40)=2.707, p=0.538), and no significant association between gender and learning outcomes (χ 2 (1, 

N=40)=0.123, p=0.726). 

The relationship between learning outcomes and the enduring time of different online learning behaviors 

To answer the research question of “what was the relationship between learning outcomes and the enduring time of 

different online learning behaviors?”, a multiple linear regression was conducted regressing the learning outcomes on 

the four predictor variables (total discussion time, total observation time, total reading time and total designing time). 

Table 6. Correlations between learning outcome and independent variables 

Intercorrelations 

Variables 

M SD 

X1 X2 X3 X4 Y 

Total discussion time (X1) 1 -0.082 -0.226 -0.624 0.319 ∗ 10.225 4.7420 

Total observation time (X2) 1 0.155 -0.143 -0.127 0.175 0.5006 

Total reading time (X3) 1 -0.535 0.566 ∗∗ 18.025 4.5825 

Total designing time (X4) 1 -0.702 ∗∗ 55.875 5.7565 

Learning outcomes (Y) 1 4.675 1.8999 

∗∗ . Correlation is significant at the 0.01 level (2-tailed). 

∗ . Correlation is significant at the 0.05 level (2-tailed). 

The correlations between the dependent variable (learning outcome) and the independent variables (total discussion 

time, total observation time, total reading time and total designing time) were shown in Table 6. The dependent 

variable was significantly correlated with total designing time (r=-0.702, p

Total 140.775 39 

2 b 

Regression 85.594 4 21.398 13.572 0.000 

Residual 55.181 35 1.577 

Total 140.775 39 

a. Predictors: (Constant), total discussion time, total reading time, total designing time. Dependent Variable: learning 

outcome. 

b. Predictors: (Constant), total observing time, total discussion time, total reading time, total designing time. 

Dependent Variable: learning outcome. 

Table 9. Results of Multiple Regression Analysis (Coefficients) 

Model a 

Unstandardized Coefficients 

B Std. Error 

Standardized Coefficients 

Beta 

t Prob. 

(Constant) 7.953 8.461 0.940 0.354 

1 

Total discussion time 

Total reading time 

0.069 

0.167 

0.108 

0.103 

0.173 

0.402 

0.641 

1.612 

0.525 

0.116 

Total designing time -0.125 0.103 -0.379 -1.219 0.231 

(Constant) 13.092 8.312 1.575 0.124 

Total discussion time 0.004 0.106 0.009 0.035 0.972 

2 Total observing time -0.969 0.423 -0.255 -2.289 0.028 

Total reading time 0.125 0.100 0.302 1.257 0.217 

Total designing time -0.189 0.101 -0.572 -1.868 0.070 

a. Dependent Variable: learning outcome. 

Findings from the multiple regression analysis were summarized in Table 7, 8 and 9. The linear regression analysis 

encompassed the individual-level variables of the subject’s learning outcome, total discussion time, total observation 

time, total reading time and total designing time. Of course, there was a precondition for model 1 and 2, that is, all 

subjects spent more than half of the total time on designing. In the first step of the analysis (Model 1), the 

simultaneous entry was specified for total discussion time, total reading time and total designing time. From table 7, 

we could find that model 1 accounted for 54.9% (R 2 =0.549) of the variance, which was significant at the 0.001 level. 

In the second step (Model 2), total observation time was added. It increased the R 2 by 5.9%. In table 8, the value of F 

was the mean square regression divided by mean square residual. The probability of the F-values in two models 

showed that the likelihood of the given correlation occurring by chance was less than 1 in 10,000. It meant that both 

linear regression equations were significant. In table 9, the values of B were the coefficients and constant of the 

linear regression equation. Beta was the B-value for standardized scores of the independent variables. The Betavalues 

indicated the relative influence of the independent variables to dependent variable. From table 9, we could 

find that, in model 1, total reading time and total discussion time had positive influence, while total designing time 

had negative influence. In model 2, total reading time and total discussion time had positive influence, while total 

designing time and total observation time had negative influence. 

Discussion 

This study explored a new and important issue on the relationship of Kolb learning styles, online learning behaviors 

and learning outcomes. It highlighted the emergent themes in following areas. Firstly, there was a significant effect 

of learning styles on total reading time and total discussion time. Convergers and Assimilators spent more time on 

online reading than Divergers and Accommodators, while Divergers and Accommodators spent more time on online 

discussing than Convergers and Assimilators. The findings were found to be theoretically consistent with the 

predictions of the Kolb learning style model. Convergers and Assimilators possessed the character of Abstract 

Conceptualization (AC). One with a high score in AC indicated that s/he was more oriented towards symbols and 

learned best in authority-directed, impersonal learning situations (Kolb, 1985). Therefore, s/he tended to read the 

electronic document of how to design the animation in the experiment. Divergers and Accommodators possessed the 

character of Concrete Experience (CE). One with a high score in CE indicated that s/he was more oriented towards 

193

peers and benefited most from discussions. Therefore, s/he tended to discuss with the graduate students who acted as 

online consultants in the experiment. 

Secondly, learning styles had no significant effect on learning outcomes. This experiment result was not anticipated 

by the researchers. However, Table 5 showed that the mean of learning outcomes of Convergers and Assimilators 

was higher than that of Divergers and Accommodators, which was in accordance with some previous conclusions. 

For example, Terrell (1995) predicted that students taking computer-mediated coursework would primarily be 

Convergers and Assimilators. Henke (2001) postulated that Assimilators and Convergers might be more successful 

to computer-based trainings than other students with different learning styles. In fact, same conclusion could also be 

drawn from the linear regression between learning outcome and the enduring time of different online learning 

behaviors. In the full model, total designing time, total reading time and total discussion time were significantly 

related to learning outcomes. Table 9 showed that either in model 1 or model 2, the standardized regression 

coefficient of total reading time was larger than the standardized regression coefficient of total discussion time. This 

meant that the influence of total reading time on learning outcomes was larger than the influence of total discussion 

time. Therefore, students who spent more time on online reading could get better learning outcomes than students 

who spent more time on online discussions. It explained the reason why the mean of learning outcomes of 

Convergers and Assimilators was higher than those of Divergers and Accommodators in this experiment. 

Thirdly, some previous studies reported a relation between gender and online learning patterns (Herring, 1992; Fahy, 

2002), but none was found in this experiment. This finding might result from the fact that computers and Internet 

access were relatively inexpensive and had become readily available in recent years. Using computers and the 

Internet was no longer seen as an exclusively or even predominantly male activity. At the university this study was 

conducted, many of the study programs had a computer literacy requirement and a degree of familiarity with 

standard computer software packages was a basic requirement for both male and female students. Students were 

accustomed to online learning environment despite gender differences; thus no significant difference was found 

between male and female students on online learning behaviors and learning outcomes. 

Finally, this study found no significant effect of learning style on total observation time and total designing time. The 

environment of the experiment design might contribute to this result. The authors of this experiment found that the 

arrangement of “online observation” was rather artificial. The subjects who were asked to “observe” the onscreen 

operation of others with their own computers might find it unhelpful to learn animation design and became reluctant 

to carry out online observation. It was likely that, in a more natural learning environment, subjects would consult 

others personally and observe what they were doing beside them, and consequently significant effect of different 

learning styles might be found. The authors of this experiment also found that the specialty of the task might be the 

reason why no significant effect of learning style was found on total designing time. In order to achieve the 

animation effects, all subjects had to spend a lot of time (more than half of the total time) dealing with the software 

of Flash. 

Conclusion 

There was a potential value in the results of the study for instructors of online courses. As was discussed earlier, 

students of Abstract Conceptualization might find that abundant electronic documents satisfied their online learning 

requirements, whereas students of Concrete Experience might find that communicative learning environments, such 

as the BBS, met their online learning demands. Based on these results, instructors using online courses should 

seriously consider the diverse learning styles when designing and developing online learning modules for different 

students. Many scholars had offered various suggestions, which included designing course modules to meet the 

requirements of observing, participation, thinking and summarizing learning circles to accommodate different 

learning styles (Simpson & Du, 2004) or offering students a learning environment that provided a variety of ways by 

which they could access course information (Ruokamo & Pohjolainen, 2000). 

In addition, maximizing students’ learning outcome through online learning was one goal of using online courses. In 

order to acquire better learning outcomes, instructors of online courses were inclined to encourage students to 

participate in online discussion activities. This study gained a different conclusion. As could be seen from this study, 

the influence of online reading played an important role in students’ learning outcome, so providing a large number 

194

of electronic documents and giving enough time to let students to absorb knowledge by online reading might also be 

effective methods to improve the quality of online courses. 

Future research 

Data analysis of this experiment proved that students belonging to different learning style types tended to have 

different online learning behaviors. It also produced the following questionable issues: Should we design online 

learning modules to meet the students’ need of different learning style types? The answers to the question might be 

arguable. For example, in Miller’s (2005) study, he claimed that understanding the compatibility of CBI (Computer 

Based-Instruction) formats for different styles allowed us to create instructional systems that were effective for all 

types of students, and CBI designers should put effort into designing systems that met the needs of all styles of 

learning/thinking. However, Robothm (1995) argued that a truly proficient learner was someone who could switch 

between styles and take advantage of all educational offerings and was someone who directed their own education. 

He believed that course design should focus on teaching students to self-direct their learning and not force students 

into a specific learning style. Taking these two different views into consideration, instructors or moderators of online 

courses should provide a variety of learning modules for students and help them learn how to switch between 

learning styles in order to take advantage of these choices. It was undoubtedly a challenging task, and would be a key 

issue of future research in distance education. 


The authors wished to acknowledge Ling-Ling Guo for her helpful assistance in the collection of the data. In 

addition, we would like to thank the students and staff who participated in this study. 

References 

Brown, E., Cristea, A., Stewart, C., & Brailsford, T. (2005). Patterns in authoring of adaptive educational 

hypermedia: a taxonomy of learning styles. Education Technology & Society, 8 (3), 77-90. 

Davidson-shivers, V., Nowlin, B., & Lanouette, M. (2002). Do multimedia lesson structure and learning styles 

influence undergraduate writing performance? College Student Journal, 36 (1), 20-31. 

Fahy, P.J. (2005). Student learning style and asynchronous computer-mediated conferencing (CMC) interaction. The 

American Journal of Distance Education, 19 (1), 5-22. 

Fahy, P.J. (2002). Epistolary and expository interaction patterns in a computer conference transcript. Journal of 

Distance Education, 17 (1), 20-35. 

Henke, H. (2001). Learning theory: applying Kolb’s learning style inventory with computer based training, retrieved 

October 15, 2007, from http://www.chartula.com/learningtheory.pdf. 

Herring, S.C. (1992). Gender and participation in computer-mediated linguistic discourse. Paper presented at the 

Annual Meeting of the Linguistic Society of America, January 9-12, 1992, Philadelphia, USA. 

Howard, W.G., Ellis, H.H., & Rasmussen, K. (2004). From the arcade to the classroom: capitalizing on students’ 

sensory rich media preferences in disciplined-based learning. College Student Journal, 38 (3), 431-440. 

Huang, R.H. (2003). The theories and Methods of Computer-Supported Cooperative Learning, Beijing: People’s 

Education Press. 

John, M.C, Pamela, A.M., & William, P.D. (1991). Factor analysis of the 1985 revision of Kolb’s Learning Style 

Inventory. Educational and Psychological Measurement, 51 (2), 455-462. 

195

Kayes, D.C. (2005). Internal validity and reliability of Kolb’s Learning Style Inventory version 3 (1999). Journal of 

Business and Psychology, 20 (2), 249-257. 

Kolb, D.A. (1985). Learning-style inventory: Self-scoring inventory and interpretation booklet, Boston: McBer and 

Company. 

Liegle, J.O., & Janicki, T.N. (2006). The effect of learning styles on the navigation needs of web-based learners. 

Computers in Human Behavior, 22 (5), 885-898. 

Melara, G.E. (1996). Investigating learning styles on different hypertext environments: hierarchical-like and 

network-like structures. Journal of Research on Computing in Education, 14 (4), 313-328. 

Miller, L.M. (2005). Using learning styles to evaluate computer-based instruction. Computers in Human Behavior, 

21 (2), 287-306. 

Oughton, J.M., & Reed, W.M. (2000). The effect of hypermedia knowledge and learning style on student-centered 

concept maps about hypermedia. Journal of Research on Computing in Education, 32 (3), 366-384. 

Raschick, M., Maypole, D., & Day, P. (1998). Improving field Instruction through Kolb learning theory. Journal of 

Social Work Education, 34 (1), 31-42. 

Reed, W.M., Oughton, J.M., Ayersman, D.J., Ervin, J.R., & Giessler, S.F. (2000). Computer experience, learning 

style, and hypermedia navigation. Computers in Human Behavior, 16 (6), 609-628. 

Robotham, D. (1995). Self-Directed Learning: the Ultimate Learning Style? Journal of European Industrial 

Training, 19 (7), 3-7. 

Ruokamo, H., & Pohjolainen, S. (2000). Distance learning in a multimedia networks project: main results. British 

Journal of Educational Technology, 31 (2), 117-125. 

Simpson, C., & Du, Y. (2004). Effects of learning styles and class participation on students’ enjoyment level in 

distributed learning environments. Journal of Education for Library & Information Science, 45 (2), 123-136. 

Smith. D. M., & Kolb. D. A. (1986). Learning Style Inventory: User's guide, Boston: McBer & Company. 

Terrell, S. (1995). Predicting success in computer-mediated coursework. Paper presented at the 6 th International 

Conference on Technology and Distance Education, October, San Jose, Costa Rica. 

Terrell, S. (2002). The effect of learning style on doctoral course completion in a web-based learning environment. 

Internet and Higher Education, 5 (4), 345-352. 

Veres, J.G. (1991). Improving the reliability of Kolb’s revised Learning Style Inventory. Educational and 

Psychological Measurement, 51 (1), 143-150. 

196

Linsey, T., & Tompsett, C. (2007). In an Economy for Reusable Learning Objects, Who Pulls the Strings? Educational 


In an Economy for Reusable Learning Objects, Who Pulls the Strings? 

Tim Linsey 

ADC (Educational Technology), London, UK // Tel: +44 20 8547 7779 // t.linsey@kingston.ac.uk 

Christopher Tompsett 

Learning Technology Research Centre, Kingston University, London, UK // Tel: +44 20 8547 7520 // 

c.tompsett@kingston.ac.uk 

ABSTRACT 

It seems a foregone conclusion that repositories for reusable learning objects (RLOs), based on common 

standards and supported by suitable search facilities, will foster a global economic market in the production of 

RLOs. Actual reuse will support producers of high-quality RLOs, and other producers will be unable to 

compete, i.e. competition within the market will implicitly define the qualities that are needed. This paper 

challenges the suggestion that this will occur. If the marked is defined as cost versus value, then the set of 

qualities that distinguishes RLOs from other educational software prohibits the development scalable search 

engines to search the repositories. At a more sophisticated level of market analysis, it is the needs of the 

producers, rather than the purchasers, that will define quality in the market. Any attempt to limit this imbalance 

will, paradoxically, require acceptance of alternative constraints that many may find hard to accept. 

Keywords 

Reusable learning object, Reconfigurability, Course design, Complexity, Learning Object Economy, Repositories 

Introduction 

Reusable learning objects (e.g., Polsani, 2003; Downes, 2003; Liber, 2005) allow the cost of initial development to 

be offset by subsequent reuse in a wide range of different contexts (Nitto et al., 2006). Conformance to a standard 

should ensure that the same RLO will function consistently in any virtual learning environment (VLE) that supports 

the same standard. As RLOs can be subdivided and recomposed in different combinations, the potential for reuse in a 

global market is significantly enhanced (Hodgins, 2002). The development of this market depends on three 

components. These are a technical and legal infrastructure to support the market, the ability to decompose RLOs to 

allow for reuse within a different learning context, and a market that supports a community of producers (Liber, 

2005) and a community of purchasers. Purchasers should be those reusing RLOs ‘at the third level’ (Koper et al., 

2004, p. 16), where there is no connection between the context of the original design and the context of reuse, other 

than the RLO itself. 

Tompsett (2005) argues that creating complex learning objects is inherently more difficult than decomposing one 

RLO into sub-components. He argues that two core problems in educational design, consistency within any set of 

RLOs and sequencing a set of RLOs, are computationally complex. In each case, small examples appear trivial to 

solve, moderately sized examples require unreasonable computing resources, and larger scale examples are 

impossible to solve. Thus, as the number of potentially useful RLOs increases and the size of the possible RLOs 

decreases, it becomes impossible to search for useable sets of RLOs, irrespective of the technological infrastructure 

or computing power that is available. 

This paper considers whether the establishment of a market in RLOs can overcome these restrictions. Educational 

designers may expect sophistication in teaching students, whereas course delivery, from an institutional perspective, 

may be more concerned with average effectiveness and economic efficiency (cf. Fletcher & Sackett, 1979). If the 

delivery of a course using RLOs is cost effective, then finding a ‘good’ educational design may be unnecessary. This 

requires, of course, that a market for RLOs exists and that a suitable set of RLOs can be discovered to match the 

economic requirements. If the conventional view (e.g. Leeder et al., 2004) holds true, then competition will drive up 

the underlying standards, to the mutual benefit of the institutions, the students and those that produce the best RLOs. 






197

This paper considers the potential for this market to exist. Firstly, taking Porter’s basic definition of a market (1985), 

the paper shows that the searches that are needed to exploit the market are computationally complex. Secondly, 

simplifying these assumptions in order to remove complexity, it then shows that this undermines what is novel in the 

RLO model. Finally, providing greater sophistication to the analysis by integrating Porter’s model of competitive 

strategy (ibid.), the paper suggests either that the market fragments, or that the producers, not the purchasers, will 

control quality. 

Background 

This argument is directed towards the design of courses that are suitably complex and contingent on the integration 

of RLOs for their design. 

A first proviso is that the courses are to be delivered within an education system in which advanced knowledge is 

delivered through institutions. The emphasis on advanced knowledge ensures that we are not dealing with relatively 

simple levels of understanding which can be constrained by additional factors (e.g. a national curriculum or standard 

assessments). The requirement that knowledge is to be delivered through institutions (e.g. Eraut, 1994, p. 118) avoids 

an outright rejection of the delivery of codified knowledge, on socio-constructivist principles (e.g. Lave & Wenger, 

1991; Brown & Duguid, 1991; Wenger, 1998). 

A second proviso is that reuse will only be considered at the third level (Koper et al., 2004) ‘at a price’. In a mature 

market, it cannot be presumed that existing objects will be reused simply because they are placed in a repository (e.g. 

Walsh, 2006). 

A third proviso is that successful course design depends explicitly on the novel, assured integration of RLOs rather 

than the linking of hypermedia resources over the Internet. Without this proviso, the discussion on RLOs would add 

little to previous work that has been achieved using Internet protocols (e.g. XML Clark, 1997; Clark & Deach, 1998) 

and hypermedia/ hypertext models over 15 years, or earlier (Nelson, 1965; Bush, 1945). 

Problems that are specific to the current market (see for example, Christiansen & Anderson, 2004) help us to 

understand why design using RLOs may be difficult now, but such problems could be considered as too detailed, or 

too specific to remain unsolved in the longer term. The discussion presumes that any transitional problems that occur 

in establishing a global market have been resolved. If the market is to provide a consistent force to increase quality, 

then the market must have acquired a degree of consistency, scale and stability for transitional effects to have 

disappeared. 

Reusable Learning Objects and Metadata 

The discussion that follows is deliberately generic, so that the interrelationships between some critical aspects are 

identified. We start the discussion with four principles, which are presented as a technical framework for RLOs in 

order to avoid arguments over particular standards. 

The first principle concerns the relationship between the technical environment and the final ‘standard’ that is 

established for RLOs. 

1. Any ‘final’ standard for RLOs will be platform neutral/independent. 

This is unlikely to be controversial since all the current standards accept this principle (cf. Johnson, 1998; 

Jesukiewicz, 2006). It provides assurance that an individual RLO will function with technical integrity on any 

platform (even if additional equipment may be required). How this will be achieved remains uncertain (Smythe, 

2004; CETIS, 2004). 

2. Integration of content is assured by conformance to standards at the metadata level. 

198

This principle ensures that ‘equivalent’ RLOs can be exchanged without losing any educational value that is covered 

by the standard. This should also be uncontroversial. Although these standards are expected to evolve, there is no 

indication that this is infeasible (CETIS, 2004; CanCore: Friesen et al., 2002; EML: Manderveld & Koper, 2004; 

SCORM: Blackmon et al., 2004, etc.). 

With these two principles in place, any set of RLOs should be interoperable, and function correctly in any new 

context, and on any technical platform, independently of the context or platform for which each of them was 

developed (IEEE, 2002). Failure to achieve this fragments the global market and limits the set of useful RLOs to 

those that are consistent with each institution’s VLE. 

The next two principles ensure that RLOs are essential to the design of a course. 

3. The standard must ensure that any RLO can be broken down into a number of smaller RLOs, or 

else is one for which any further decomposition fails to make sense (either technically, or 

educationally). 

This principle distinguishes RLOs from other existing educational resources. It is assured by the use of XML, 

although, for this analysis, there is no requirement that the decomposition is hierarchical. This principle establishes a 

key property: each fragment is less tightly bound to the context of its original development (Hodgins, 2002) and, as a 

consequence, the market will contain a large number of RLOs for which the context of reuse is under-specified. 

Finally we require that: 

4. It must be possible to identify relevant RLOs to integrate into a course on the basis of searchable 

meta-data. 

This is inherent in any of the current proposals as meta-data is included within each RLO, but a wider range of 

solutions could be envisaged. The critical issue is that the potential value of any RLO can be represented as static 

information. If a full assessment requires that an RLO be inspected in detail, then it cannot be claimed that 

integration between RLOs is achieved simply through the use of RLOs, rather than through some additional factor 

that is intricately ‘designed in’ to individual RLOs but which is not coded in the meta-data. 

These four principles should be sufficient to ensure, given a suitable technical and legal infrastructure (e.g., Downes, 

2003; Koper et al., 2004, et al.), that a global market for RLOs could exist in which: 

• a search on meta-data across repositories (e.g. van Assche et al., 2006), or meta-repositories, would identify any 

possible RLOs that could be used in constructing a course from RLOs. 

• individual RLOs that have been ‘purchased’ can be integrated into more complex RLOs without the need to test 

that the combination will function as expected. 

At this point the concept of ‘course design’ is deliberately left unrefined. Although many argue that current models 

are limited (e.g. Kassahun et al., 2006), a more critical issue is whether the market will be of sufficient size to be 

self-supporting. It is suggested that four assumptions are essential. It will only be necessary to consider what is 

meant by course design using RLOs if such a market is viable. 

A market for RLOs? 

Few authors have considered that a market for RLOs might not exist. To clarify this, four critical assumptions are 

identified on which the rest of the argument is structured. These should be uncontroversial. 

The first two ensure that the market is of a sufficient size to allow a choice for most courses to be designed using 

RLOs. 

1. Almost all searches for a single RLO, with some search criteria unspecified, will produce multiple solutions. 

2. Almost all searches for a single, fully specified RLO, which could be used as a self-sufficient component of a 

course, will fail. 

199

The first assumption ensures that creating a course from existing RLOs will not fail because specific components do 

not yet exist. The second condition reflects the fact that the majority of components are under-specified. The second 

condition also excludes the utopian scenario in which any course could be constructed by selecting a handful of 

learning objects. If this were to be allowed, then it becomes impossible to argue that the success of any course 

depended on design by RLO, rather than on detailed interrelationships within each RLO. 

Once RLOs are available on this scale, then the simplest ‘market’ principles must also apply. As Porter notes: 

“Buyers must be willing to pay a price for a product that exceeds its cost of production, or an industry will not 

survive in the long run” (1985, p. 8). Since the market must reward those that produce ‘good’ RLOs, two further 

assumptions are introduced. 

The first is: 

3. Purchasers must be willing to pay a price for each RLO that allows the producers of good quality RLOs to make 

a reasonable profit in the long run. 

The ‘cost’ to the institution for using the RLO will need to include this price, together with predicted estimates for 

technical, teaching and administrative support. The final assumption reflects the reciprocal nature of the market. If a 

purchaser buys any RLO from the market, then they must be able to associate an ‘educational value’ (EV) to each 

possible purchase without detailed inspection of the RLO. This leads to the final assumption: 

4. Every purchaser must be able to give an estimated EV for each RLO that is calculated on the meta-data 

available. 

For simplicity EV is assumed to be numerical. Within the argument that follows, there is no need for EV to be 

expressed in financial terms, nor is there a need to be more precise about how this is calculated. More complicated 

evaluations of individual RLOs will only exacerbate the decision problem that follows. 

The set of principles and assumptions listed above should be sufficient to allow each purchaser to select a set of 

RLOs in order to construct a course from the global market of RLOs, in which technical and educational 

effectiveness is assured. 

If the market exists, then the market will genetically develop a set of qualities that are, de facto, those that ensure 

competitive success. The critical question is whether these qualities will be consistent with ‘fitness for course 

delivery’ from an institutional perspective, or whether other forces in the market will generate an alternative set of 

qualities. 

Collecting sets of RLOs and the knapsack problem 

To assess this argument the economic decision must be analysed in more detail. The fundamental question is: can a 

cost effective set of RLOs be found whilst only considering the cost and EV of each RLO? 

This is implicitly a question of reconfigurability: “the potential of a collection of existing RLOs to be re-configured 

as a larger, educationally effective part of a course and to integrate with that course” (Tompsett, 2005, p. 446). In the 

original paper, two educational properties of a set of RLOs were identified, each of which required a search through 

repository based properties of individual RLOs. Both of these problems were shown to belong to a wider set of 

decision problems, that are termed ‘Non-Polynomial Complete’ (NPC, Garey & Johnson, 1990). in order to establish 

a subset to meet property of the complete set. All NPC problems are equally complex to solve and none are possible 

to solve in practical terms, irrespective of the computing power that is available (or the speed of any network). The 

first critical point is to show that this market problem is indeed an NPC problem. 

The search that is needed can be modelled as a ‘knapsack problem’, which belongs to the same set. A knapsack 

problem, with minor rephrasing from Garey & Johnson (1990, p. 65), requires that: 

“A number of objects can be taken on a journey. Each of them has a certain value and occupies a 

certain size in the rucksack - we do not worry, for now, whether they would actually fit together. The 

problem is to decide if a subset of these can be found that will: 

200

e below a maximum size (of the knapsack), but allow you to take at least a certain 

amount of value.” (italics as original) 

If we draw a parallel between value and EV, and between size and cost, then the course designer is faced with the 

following problem: 

A number of RLOs can be purchased to use on a course. Each of them has a certain EV and can be 

purchased at a given cost - we do not worry, for now, about other criteria. The problem is to decide if 

any subset of the RLOs can be found that will: 

have a total cost within maximum budget for the course but offer at least a certain 

amount of EV. 

From an institutional perspective, it is impossible for the course designer to search across the global market in order 

to select an effective choice from the full set of RLOs. As the number of useable RLOs increases, and the size of the 

possible RLOs decreases, then it becomes infeasible to collect economically useful sets of components, irrespective 

of the technological infrastructure, computing power, or standards that are established. State of the art algorithms 

(see Tompsett, 2005) fail; ignoring ‘other criteria’ invalidates the approximations that are applied in the ‘best’ 

algorithms (see Appendix A). The only alternative is to place an upper limit on the number of components that can 

be selected - contradicting principle 3. 

At this point we reach a central point of the paper. If the market for RLOs conforms to the principles and 

assumptions that have been outlined above, then a global search cannot be supported by software, either locally, or 

centrally within repositories. The ‘mathematical’ complexity of reconfigurability will persist irrespective of changes 

in coding such as those envisaged by the introduction of the semantic coding (Tompsett, 1991), the semantic net 

(Dzbor et al., 2007) or the introduction of peer-to-peer technology (Brito & Moura, 2005). 

Contrary to expectations, fragmenting RLOs into an increasing number of smaller components will reduce the 

possibility that the purchaser can make an effective decision. 

In order to circumvent this problem, we will either need to remove the ‘mathematical’ complexity from the decision 

problem or add more detail to the analysis of the market in the hope that additional detail will simplify the search (cf. 

Waltz, 1975). 

Simplifying the market 

Although scale is the key factor that controls the size of the search, the inherent complexity of this problem is created 

by the ratio between the EV and the cost of each RLO. Simplifying the EV to cost ratio (EVCR) will remove 

complexity from the problem. Two classes of simplification: minimal-cost and cost-related-to-value are considered 

below. 

Minimal-cost covers scenarios in which the cost of course design using RLOs is so low that any other approach to 

course delivery would be dismissed. A universally low cost model covers any scenario in which the price of every 

RLO is extremely low. This must be dismissed as irrelevant to improving quality. If the market is large, then this 

scenario treats the actual decisions as irrelevant and removes any possible influence that these decisions will have to 

improve the quality of the products that are available. Altruism, an alternative to universally low cost, would suggest 

that a suitably large number of RLOs are made available to the market at an artificially low cost, as the development 

is benevolently funded from external sources. At the first level of analysis this sounds ideal for both the purchaser, 

who has little financial risk, and the producer, who has assured development costs. However, it is not the consumer 

that is directing the market in this case: the critical choices are made by those who fund the development. Any 

developer that attempts to compete on the ‘open market’, without initial funding, is immediately placed at an 

economic disadvantage and so competition is no longer ‘open’. 

This problem exists even if funding is restricted to the early stages of market development. Assured funding allows a 

small number of producers to establish a number of RLOs within the market of ‘early adopters’. These will provide a 

funding stream to finance future work – an advantage which is even more critical if the price of the existing RLOs is 

201

allowed to rise when external funding is removed. With established teams and procedures, these producers will be 

able to provide new RLOs to the market with lower overheads for production and financing. These producers can 

then choose their own strategy to limit any ‘new entrant’ producers (Porter, 1985, p. 131). This could vary, from full 

exclusion – offering new RLOs at costs that cannot be matched by new competitors, to strategic selection of new 

competitors in order to provide a ‘cost umbrella’ to increase profitability (‘competitor selection’, 1985, p. 201 ff.). 

Such strategies are only vulnerable if a significant number of purchasers collaborate to widen the range of producers, 

and accept the current cost disadvantage as part of a longer-term strategy to broaden the market. Even if the costs for 

these funded RLOs are not allowed to rise, the market penetration, almost certainly in the most re-useable RLOs, will 

give these funded producers significant opportunities to select one of a wide range of market strategies (see below) to 

control the sections of the market with the highest profitability. 

Cost-related-to-value characterizes a range of solutions in which the cost is constrained (or calculated by methods 

that do not introduce further complexity) to fall within specific limits of the EV, or vice-versa. In such cases, only 

one of the factors needs to be considered as the control factor in selecting a set of RLOs. In the simplest version, the 

cost of RLOs with equivalent meta-data must be the same and it is tempting to view this as a scenario that sets 

uniformly high standards. Unfortunately it is equally possible that all RLOs are as ineffective as each other. In the 

more general case, the cost of any particular RLO must lie between a maximum and minimum value, both of which 

can be calculated from the EV. However, the producers will then lose the ability to differentiate between their own 

RLOs. With a maximum and minimum possible cost for each RLO, competition between producers will produce 

RLOs to either one or both of these costs (e.g. as is currently the case with ‘top-up fees’ in UK universities). 

Competitiveness between producers is then based on minimizing the cost of production to achieve fixed standards. 

This divides the market and, unless any other strategy is in place, forces each producer to adopt a 'cost leadership' 

strategy (minimizing production costs: Porter, 1985, p. 11; economies of scale: Shapiro & Varian, 1999, p. 25). 

Whilst this is apparently attractive to the purchaser, the scenario decreases quality overall. The producer must set out 

to achieve as small a reduction in the cost to the purchaser as necessary, whilst minimizing their own production 

costs compared with other producers. The benefits to the purchaser are minimal in the short term and contradictory in 

the long term. As Porter notes, when more than one producer adopts this strategy, the market becomes unstable. The 

producers undercut each other and the overall quality of what is produced is pushed down in order to maintain a 

viable level of profit. The only positive outcome for the purchaser is the development of a new market for cheap, 

low-end products with unreliable quality. Alternatively, both quality and margins continue to be cut until a single 

producer remains. At this point the cost for the low-end product can increase without any competition to ensure that 

quality will also increase. Cost-related-to-value does not improve quality from the purchaser’s perspective (cf. Tesco 

Supermarket in the UK). 

Neither of these simplifications: minimal-cost or cost-related-to-value, produces an economic market in which the 

purchaser controls quality. Both, however, raise questions about the nature of the relationships between producers, 

and between purchasers and producers. 

Improving the market model 

It still remains plausible to suggest that a more detailed analysis of the market, providing more economic reality, will 

remove complexity, without the purchaser losing control over quality. 

In order to do so we must move to a model in which each producer adopts a strategy for competition within the 

market. Porter’s model identifies four strategies that can be used by a producer of RLOs to compete with others in 

order to gain, or protect market share. Despite the changes that might be considered to have changed in a networked 

world, the economic principles of these markets remain remarkably similar (Shapiro & Varian, 1999, p. 2). Two 

strategies act globally, lowest-cost and brand-name. The other two, focus and differentiation, segment the market. 

Cost leadership has already been discussed and shown to undermine, rather than improve quality. A brand-name 

strategy is considered next. 

A brand-name strategy builds an image of high quality, in direct contrast to the lowest-cost strategy. The strategy is 

based on trading at a higher-than-market price, on the basis of an apparent value, even if there may be no actual 

202

increase in quality. In the most successful cases, the brand name can become a de facto justification of a good 

decision (cf. ‘no-one ever got sacked for buying IBM’). When written in its simplest form, this makes clear that the 

purchaser is paying a cost for the illusion of quality, rather than for quality itself. More realistically this approach 

exploits the difference between indicators of quality, rather than quality itself. If it is more efficient to create the 

impression of higher value, rather than to build higher quality into the RLOs, then a higher level of profit can be 

maintained. The producer may choose to invest this profit into promoting the brand name, or into improving quality, 

but the customer cannot affect this decision. Unless the reputation is established, in the first place, by a de facto 

higher quality relative to cost for the purchaser, and the higher level of profit is used to maintain this ratio relative to 

other competitors, the effect on the market is to reduce quality relative to cost. This strategy is naturally restricted to 

a small number of producers, and the value of any margin is expected to be less marked for information, as opposed 

to material products (Shapiro & Varian, 1999, p. 31). 

A strategy based on focus or differentiation, or any combination of the two, responds to differences between 

purchasers. Focus responds to characteristics that are inherent in the market (Porter, 1985, p. 131) and concentrates 

on the delivery of products to that sector. This allows internal processes to be optimized to produce a narrower range 

of products where optimization would not be cost-effective within a global market. For any purchaser within that 

sector, the scale of the search is proportionately reduced, but these linear effects have no more impact on the 

effectiveness of the search process than increasing the speed of the underlying computers. Such changes have 

negligible effect on complexity unless the actual number will always remain low – i.e. the customer has fewer 

options and less choice. The effect on other sectors is minimal. If purchasers are to control quality, then this can only 

come from the one remaining approach: differentiation. This final strategy is the most complex and is considered in 

more detail. 

Differentiation requires that “a firm be uniquely able to create competitive advantage for its buyer in ways besides 

selling to them at a lower price” (Porter, 1985, p. 131). This takes into account factors that are specific to the 

competitive strategy of each purchaser, rather than those that are common across a sector. If the supplier of an RLO 

can design their product to meet the needs of a producer and increase the profit margin of the producer in their own 

product, then the producer of the RLO can demand a higher price for the RLO relative to competitors’ RLOs. This 

depends on the ‘use criteria’ for each purchaser, i.e. the qualities that allow each one to differentiate themselves 

within their own market (Porter, 1985, p. 137). If an institution can increase their profit margin on a particular course 

by selecting one producer of RLOs, rather than another, then it is cost-effective to pay more than the lowest-cost 

price for an equivalent RLO. This is a symbiotic relationship. Both the producer of the RLO and the course provider 

must increase their profits, and protect this increase, a form of economic lock-in (Shapiro & Varian, 1999, p. 110 ff.) 

We shall return to this point later. 

The most obvious examples occur at the lower end of the purchaser’s value-chain, for example by customizing each 

RLO to the specific needs of the purchaser at a cost that is lower than would be incurred by the purchaser. Less 

obvious examples occur where the value is added directly to the upper end of the purchaser’s value-chain (cf. ‘intel 

inside’). Porter discusses a number of standard approaches to differentiation, of which three (summarised from p. 

135) could be seen as relevant to the market for RLOs. Expressed in terms of how these affect the value-chain for the 

delivery of a course, these would be: lowering the direct cost of teaching time, lowering the cost of support staff, and 

lowering the risk of failure. Although each of these seems indicative of an increase in quality, a more detailed 

analysis needs to be taken to understand the impact of this effect on the global market. 

A reduction in teaching time can occur either through reducing the time that is spent on integrating an RLO into a 

course, or by directly reducing the time that is needed, e.g. with automated assessment. Reducing the cost of 

integration with part of the course that is not supported by an RLO suggests that a ‘gap’ exists within the market 

(contradicting assumption 1). Suggesting that there is a reduction in the cost of integration between RLOs indicates 

that cost-effective design depends on more than metadata (contradicting principle 4). Reducing the cost associated 

with running an RLO is evidently advantageous for that particular institution and, as the relationship is designed to 

be of mutual benefit, differentiation circumvents the search process for that institution. However, within the global 

market, the effect is minimal. The cost of an RLO may be reduced, but the RLO must still be discovered through the 

search process by other institutions. The changes will only have an effect on the wider market if the reduced cost 

reduces the complexity of the search – which is not the case. Since there is no reduction in complexity within the 

global market, there is no advantage, unless the same improvements can be copied by all the producers (see below). 

203

The second option, lowering the cost of support staff (and facilities) is unlikely to be significant. These costs are 

almost certain to depend on the underlying technology, and not the RLO itself (principles 1, 2 and 3). Switching 

costs should be minimal and reduce the risk of hardware/software lock-in (Shapiro & Varian, 1999, p. 116). In 

specialised fields, some differentiation may occur. For simple anatomy, the low-cost, ubiquitous, ‘Anatomy 

Colouring Book’ (Kapit & Elson, 2001) may require far less technical support than a virtual reality model to do the 

same. However, if the technical infrastructure is available to support a virtual model, then the same technology can 

be used in a wider set of courses (Székely & Satava, 1999). However, such cases are limited and will have little 

impact on the overall standards that are established. 

The final consideration, lowering the risk of failure, might appear to act as a discriminating factor at the institutional 

level but is implicitly limited by the design of RLOs. If RLOs conform to the standards, there should be no risk of 

technical failure (principle 1), or lack of integration with other RLOs (principle 2). Neither should there be any 

possibility of failure in educational terms, as long as the meta-data is valid (principle 4). The only scope for increased 

differentiation in this category is that some RLOs will lower the probability of failure in educational terms by factors 

that cannot be determined from the meta-data. We suspect that this may be the most interesting aspect – but that 

contradicts principle 4. 

However, before we close the discussion on differentiation, we must note that the symbiotic relationship provides an 

additional limit to the effect that differentiation can have on a global market. As noted above, an increase in profit 

must be protected for both the producer of the RLOs and the particular institution that is involved. If other producers 

of RLOs can imitate the same changes in their products, then competition is reintroduced and the added value of the 

specialization for the producer decreases, or disappears. The same effect applies to the purchaser’s market. If every 

course provider can achieve the same benefit, then the increase at the upper-end of the value-chain is lost and the 

higher price for each RLO cannot be financed. This implies that both the producer and the purchaser must prevent 

imitation, through an effective rights management strategy (copyright and patent protection, etc.) Actual ‘bitlegging’, 

as Shapiro and Varian comment, is naturally limited; the more effective a ‘bitlegger’ is at advertising, the more 

easily they are discovered (1999, p. 92). 

Power to the purchaser? 

The introduction of a more sophisticated model of the global market suggests that, in almost all cases, the 

institutions, when acting as independent purchasers, will have only minimal effect on the standards and underlying 

quality of the RLOs. Only differentiation, as a competitive strategy for the producer, suggests that competition leads 

to improvements from an institutional perspective. Even then, the institutions and producers involved must act to 

prevent, rather than support, a wider increase in quality. 

Conclusions 

This paper reviewed the proposition that good quality would become defined and controlled by the purchasers within 

a global market for RLOs. On the basis of a minimal set of principles and assumptions, it was argued that the 

economic decision to select an economically efficient set of RLOs creates a search problem that is impossible to 

support in software. 

This model of the market was then reviewed in two ways. Simplifying the model, in order to reduce complexity, 

either lowers standards, or allows external agencies rather than educational institutions to define the qualities that are 

needed. Providing a more detailed analysis of the market cannot reduce the complexity of the search process and, if 

anything, suggests that the institutions will have even less control over quality. Even though quality might be 

improved in a limited number of instances, market forces will act to retain distinctions between producers, rather 

than to promote consistent improvement across the market. Whilst standards in RLOs emphasize uniformity, the 

economics of the market place react to foster diversity between producers and to fragment the market. 

Two issues offer scope for a deeper analysis than is possible here. The first is that both ‘cost’ and ‘educational value’ 

have been left under-specified. This was chosen to emphasize that the argument holds true irrespective of whether 

204

these were defined on a global, or local, basis. Contextualising these definitions for each institution will, almost 

certainly, increase complexity but allow the possibility that quality may still be achieved, from a local perspective, 

even if the global market cannot be controlled in this way. However, that would suggest that the concept of coursedesign 

based on RLOs is contingent on local, non-generic effects. The inter-relationship between market structure, 

market strategy and context-dependent value will need much more analysis if quality is to imply anything more 

useful to course designers than conformance to technical standards. 

The second issue to explore is the importance of signalling criteria (Porter, 1985, p. 142), indirect indicators used by 

a purchaser to identify ‘good’ producers (cf. institutional reputation in filtering a literature search). Signalling criteria 

cover the additional information, beyond product specification, that is used to build trust between purchasers and 

producers that each RLO will perform ‘as specified’. Assumption 4 may therefore need to reflect this more directly. 

However, although collection and management of this information is well-understood within recommender systems 

(Resnick & Varian, 1997), as in MERLOT (2003), the parallel management and control of such information from the 

producers’ perspective would also need to be considered. 

Even so, it should be clear that a global market provides no quick solution to defining quality from a purchaser’s 

perspective. This is a new facet of reconfigurability (Tompsett 2005) - integrating individual RLOs into a more 

complex RLO is categorically harder than taking a complex RLO and fragmenting it into a set of simpler RLOs. 


The authors are grateful for general editorial comment from Dr. Linda Burke and Hilary Tompsett and for a critical 

appraisal of the economic arguments from Dr. R. Roberts of Kingston University. We are also grateful to the critical 

comment from both reviewers of the original submission. These have offered the opportunity for some of the 

arguments to be elaborated and others to be made more precise. 

References 

Blackmon, W., Brooks, J., Roberts, E., & Rehak, D. (2004). The Overlap and Barriers between SCORM, IMS Simple 

Sequencing, and adaptive Sequencing, retrieved October 15, 2007 from http://141.225.40.64/lsal/expertise/papers/ 

conference/ah2004/wit20040329.pdf. 

Brito, G. A. D. D., & Moura, A. M. d. C. (2005). ROSA - P2P: a Peer-to-Peer System for Learning Objects 

Integration on the Web. In R. P. M. Fortes (Ed.), Proceedings of the 11 th Brazilian Symposium on Multimedia and 

the web, New York: ACM Press, 1-9. 

Brown, J. S., & Duguid, P. (1991). Organizational learning and communities of practice: towards a unified view of 

working, learning and innovating. Organization Science, 2 (1), 40-58. 

Bush, J. V. (1945). As We May Think. The Atlantic Monthly, 176 (1), 101-108. 

CETIS (2004). The e-learning Framework Summary June 2004, retrieved October 15, 2007, from 

http://www.jisc.ac.uk/uploaded_documents/elf-summary7-04.doc. 

Christiansen, J.-A., & Anderson, T. (2004) .Feasibility of Course Development Based on Learning Objects: Research 

Analysis of Three Case Studies. International Journal of Instructional Technology and Distance Learning, March, 

retrieved October 15, 2007, from http://www.itdl.org/Journal/Mar_04/article02.htm. 

Clark, J. (1997). Comparison of SGML and XML, retrieved October 15, 2007, from http://www.w3.org/TR/NOTEsgml-xml-971215. 

Clark, J., & Deach, S. (1998). XSL version 1.0, retrieved October 15, 2007, from http://www.w3.org/TR/1998/WDxsl-19980818. 

205

Downes, S. (2003). Design and Reusability of Learning Objects in an Academic Context: A New Economy of 

Education? United States Distance Learning Association Journal, 17 (1), 3-22. 

Dzbor, M., Stutt, A., Motta, E., & Collins, T. (2007). Representations for semantic learning webs. Journal of 

Computer Assisted Learning, 23 (1), 69-82. 

Eraut, M. (1994). Developing professional knowledge and competence, London: Falmer. 

Fletcher, S., & Sackett, D. (1979). The periodic health examination: Canadian Task Force on the Periodic Health 

Examination. Canadian Medical Association Journal, 121, 1193-1254. 

Friesen, N., Roberts, A., & Fisher, S. (2002). CanCore: Metadata for Learning Objects. Canadian Journal of 

Learning and Technology, 28 (3), retrieved October 15, 2007, from http://www.cjlt.ca/content/vol28.3/ 

friesen_etal.html. 

Garey, M. R., & Johnson, D. S. (1990). Computers and Intractability; A Guide to the Theory of NP-Completeness, 

New York: W. H. Freeman & Co. 

Hodgins, H. W. (2002). The future of learning objects, retrieved October 15, 2007, from 

http://www.reusability.org/read/chapters/hodgins.doc. 

Horowitz, E., & Sahni, S. (1974). Computing Partitions with Applications to the Knapsack Problem. Journal of the 

ACM, 21 (2), 277-292. 

IEEE (2002). LOM Data Model Standard (1484.12.1), retrieved October 15, 2007, from http://ltsc.ieee.org/wg12/. 

Jesukiewicz, P. (2006). Sharable Content Object Reference Model (SCORM): An Overview and Update for HPT 

Professionals, retrieved October 15, 2007, from http://www.adlnet.gov/downloads/DownloadPage.aspx?ID=238. 

Johnson, D. (1998). ADL Initiative Overview, Tasking and Direction Briefing, retrieved October 15, 2007, from 

http://adlnet.gov/downloads/downloadpage.aspx?ID=15. 

Kapit, W., & Elson, L. M. (2001). The Anatomy Colouring Book, Wokingham: Addison Wesley. 

Kassahun, A., Beulens, A., & Hartog, R. (2006). Providing Author-Defined State Data Storage to Learning Objects. 

Education Technology and Society, 9 (2), 9-32. 

Koper, R., Pannekeet, K., Hendriks, M., & Hummel, H. (2004). Building communities for the exchange of learning 

objects: theoretical foundations and requirements. ALT-J, 12 (1), 21-35. 

Lave, J., & Wenger, E. (1991). Situated Learning: Legitimate peripheral participation, Cambridge: Cambridge 


Leeder, D., Boyle, T., Morales, R., Wharrad, H., & Garrud, P. (2004). To boldly GLO - towards the next generation 

of Learning Object. Paper presented at the E-Learn 2004 Conference, 1 - 5 November 2004, Washington, DC, USA. 

Liber, O. (2005). Learning Objects: conditions for viability. Journal of Computer Assisted Learning, 21 (5), 366-73. 

Manderveld, J., & Koper, E. J. R. (2004). Educational modelling language: modelling reusable, interoperable, rich 

and personalised units of learning. British Journal of Education Technology, 35 (5), 537-551. 

MERLOT (2003). Multimedia Educational Resource for Learning and Online Teaching, retrieved October 15, 2007, 

from http://www.merlot.org/. 

206

Mitchell, G. G., O'Donoghue, D., & Trenaman, A. (2000). A New Operator for Efficient Evolutionary Solutions to 

the Travelling Salesman Problem. Applied Informatics 2000, Innsbruck, Austria, retrieved October 15, 2007, from 

http://www.cs.may.ie/~georgem/papers/TSP_AI_2000.ps. 

Nelson, T. H. (1965). A File Structure for the Complex, the Changing and the indeterminate. Proceedings of the 

1965 20 th national conference, New York: ACM Press, 84-100. 

Nitto, E. D., Mainetti, L., Monga, M., Sbattella, L., & Tedesco, R. (2006). Supporting Interoperability and 

Reusability of Learning Objects: The Virtual Campus Approach. Education Technology and Society, 9 (2), 33-50. 

Polsani, P. R. (2003). Use and Abuse of Reusable Learning Objects. Journal of Digital Information, 3 (4), retrieved 

October 15, 2007, from http://jodi.tamu.edu/Articles/v03/i04/Polsani/. 

Porter, M. E. (1985). Competitive Advantage, New York: The First Free Press. 

Resnick, P., & Varian, H. R. (1997). Recommender Systems. Communications of the ACM, 40 (3), 56-58. 

Shapiro, C., & Varian, H. R. (1999). Information Rules: A strategic guide to the networked economy, Boston, MA, 

USA: Harvard Business School Press. 

Smythe, C. (2004). State-of-the-Art Report on Technologies and Techniques for Testing, retrieved October 15, 2007, 

from http://www.opengroup.org/telcert/documents/TELCERT_State_of_the_Art_Report.pdf. 

Székely, G., & Satava, R. M. (1999). Virtual Reality in Medicine. BMJ, 319 (13 November), 1305, retrieved October 

15, 2007, from http://www.bmj.com/cgi/content/full/319/7220/1305. 

Tompsett, C. (1991). Contextual Browsing in a Hypermedia Environment. In Giardina, M. (Ed.), NATO Invited 

International Workshop on Interactive Multimedia, New York: Springer Verlag, 95-110. 

Tompsett, C. (2005). Reconfigurability: creating new courses from existing learning objects will always be difficult! 

Journal of Computer Assisted Learning, 21 (6), 440-448. 

van Assche, F., Duval, E., Massart, D., Olmedilla, D., Simon, B., Sobernig, S., Ternier, S., & Wild, F. (2006). 

Spinning Interoperable Applications for Teaching & Learning using the Simple Query Interface. Education 

Technology and Society, 9 (2), 51-67. 

Walsh, K. (2006). Reusable Learning Objects. BMJ, 332 (20 May), 1193, retrieved October 15, 2007, from 

http://www.bmj.com/cgi/content/full/332/7551/1193-a. 

Waltz, D. (1975). Understanding line drawings of scenes with shadows. In Winston, P. H. (Ed.) The Psychology of 

Computer Vision, New York: McGraw-Hill, 19-91. 

Wenger, E. (1998). Communities of Practice: Learning, Meaning and Identity, Cambridge: Cambridge University 

Press. 

207

Appendix A. Good-enough approaches 

The standard method to solving knapsack problems follows the approach developed by Horowitz & Sahni (1974; see 

also Mitchell et al., 2000). In purchasing a set of RLOs, the organisation is assumed to be able to estimate a target 

ratio (EVCR) between the overall cost to run the course and the overall EV (whatever the mode of delivery). This 

ratio can then be used to remove from initial consideration any individual RLOs with a ratio that is much worse than 

this target value. This reduces the size of possible RLOs to search through, though it does not alter the number of 

RLOs that could be included in a solution. Once a set of potential solutions has been found, a check can be 

conducted to test whether swapping any of the current ‘best set’ with some of those that were ignored, would 

produce a better solution overall. 

The validity of this approach depends on the sets of RLOs that are collected on the basis of the EVCR value. A 

simple example shows that estimating the EV of a combined set is far harder than this. This example is ‘trivial’ in 

scale but should suffice to illustrate why the standard algorithm would never work. 

Table 1. Selecting resources 

Resource Coverage EV Cost EVCR 

A a, b, c, d 8 8 1 

B a, b 3.6 4 0.9 

C e, f 2.8 4 0.7 

Table 1, above, shows the coverage, EV and cost for three resources. The current ‘best-approach’ model would 

select the first two resources (A and B) in order to produce the best set with a budget of ‘12’. Swapping C for B 

could never be considered as it has the lowest nominal EVCR. Ignoring the other information, i.e. coverage of each 

resource, misses the duplication of material that has occurred. 

208

Knowlton, D. S. (2007). I Design; Therefore I Research: Revealing DBR through Personal Narrative. Educational Technology & 

Society, 10 (4), 209-223. 

I Design; Therefore I Research: Revealing DBR through Personal Narrative 

Dave S. Knowlton 

Department of Educational Leadership; Southern Illinois University Edwardsville, USA // Tel: +1 618.650.3948 // 

Fax: +1 618.650.3808 // dknowlt@siue.edu 

ABSTRACT 

Design Based Research (DBR) is a new and still-emerging approach to research about design, learning, and 

allied areas. This article reports one designer’s experiences within a DBR project. Whereas most reports of DBR 

focus on the outcomes of the design itself, the current paper offers a hermeneutical perspective by focusing on 

the personal narrative of the designer. Using Edelson’s (2002) categories of learning as a basis for the 

discussion, the author reports the development of his own domain theories, design frameworks, and design 

methodology. Implications for other designers who would consider using personal narrative and hermeneutics 

are offered. 

Keywords 

Design-based research, Rapid prototyping, Personal narrative, Computer-mediated bulletin boards, Theory 

development 

Introduction 

Because design-based research (DBR) is a relatively new and still-emerging methodology that is used by learning 

scientists, we have much to learn about its processes and resulting products. Most scholarship about design research 

has focused on the outcomes, such as the ontological innovations, that emerged from a DBR experiment (cf., diSessa 

& Cobb, 2004). This is as it should be since one of the main points of DBR is to improve educational practice 

(Collins, Joseph, & Bielaczyc, 2004). Design, however, can be a valuable learning experience in its own right 

(Edelson, 2002; Knowlton, 2004a; Nelson, 2003; Nelson & Knowlton, 2005; Wiggins & McTighe, 2005), and even 

when designers are not acting as researchers, per se, their own reflections on their efforts within a design project can 

be useful toward a more complete understanding of design-based research as a phenomenon of engagement. 

Therefore, designers should examine their experiences as designers and describe the ways that they find themselves 

situated within design scenarios. One way for designers to illustrate their own situatedness within various scenarios 

is through personal narratives. Personal narratives can capture the experiences of designers within specific contexts, 

and thus create a shift from curriculum and pedagogical development towards DBR, a shift described and justified by 

Edelson (2002). But this type of DBR focuses on the interactions between a designer and a design, rather than the 

outcomes of design. After all, as noted by Barab and Squire (2004), the way that designers are situated within a 

context can impact the research itself. 

The purpose of this paper is to offer my personal narrative as a designer charged with prototyping strategies to 

support the use of a computer-mediated bulletin board among university students. In the first part of this paper, brief 

support for rapid prototyping as an appropriate design methodology within higher education is provided. In the 

second part of this paper, I describe the particular design efforts on which this paper is based. These first two parts of 

the paper serve as a contextual frame for supporting the third part of this paper, in which I offer a personal account of 

my design experiences. In the fourth part of this paper, my experiences are generalized in an effort to offer 

implications for other would-be designers who may find value in considering design scenarios through a personal 

narrative approach. 

In its totality, this paper is based in the exploratory research tradition (Marshall & Rossman, 1995), in that this paper 

serves “to investigate little understood phenomena to identify/discover important variables to generate hypotheses for 

further research” (p. 41). It also is framed by hermeneutics in that it "establishes context and meaning” for human 

action (Edelson, 1988; Patton, 1990, p. 85), and it attempts to provide an understanding of the contextualized “lived 

experience" (O'Grady, Righy, & Van den Hengel, 1987, p. 94) of a designer. 






209

Rapid Prototyping and Design Based Research 

Formal instructional design often is too expensive of a process to be viable in the higher education classroom on a 

day-to-day basis; and carefully-designed learning environments require such detailed planning that it is unrealistic to 

expect college professors to enter each semester with a solid design that fully supports an inquiry-based or 

exploratory environment. Sometimes, then, the best a faculty member can do is to prototype an assignment—to 

design “something” as a part of a new course preparation and tweak it over time through iterative cycles of 

implementation and revision. Reiser (2001) notes that rapid prototyping is established as a widely-used process 

because it allows a designer to arrive at instructional products more quickly. In this respect, rapid prototyping is also 

appropriate as a basis for design research in that design research is based on “progressive refinement”—“putting a 

first version of a design into the world to see how it works” and then revising the design “until all the bugs are 

worked out” (Collins, Joseph, & Bielaczyc, 2004, p. 18). Thus, rapid prototyping has dual potential in that it can 

serve both as a practical approach to curriculum and pedagogical development and as a basis for scholarly inquiry. 

In spite of this dual potential for productivity in higher education, the use of rapid prototyping in higher education is 

not well documented in academic literature. One explanation for this deficiency in the literature is that rapid 

prototyping is a process more closely associated with meeting goals that are indicative of business and industry 

(Stokes & Richey, 2000). Thus, higher education faculty members simply might overlook rapid prototyping as a 

viable design approach that is worthy of being documented. Even when those in higher education recognize rapid 

prototyping as a viable design approach, documentation efforts may be hindered because models of rapid prototyping 

are surprisingly complex (Stokes & Richey, 2000); and the sound execution of rapid prototyping often requires a 

support team, such as graduate assistants and experienced designers (cf., Lohr, Javeri, Mahoney, Gall, Li, & 

Strongin, 2003). The complexity of process and lack of available human capital to support such a process may cause 

faculty members to shy away from documenting their efforts. 

Furthermore, within most disciplines, scholarship that deals with pedagogy, curriculum development, and allied 

areas is not valued as highly as discipline-based research. Even where valued, reporting ideas from implementing a 

design like rapid prototyping requires a reconsideration of research paradigms. As Barab and Squire (2004) note, 

DBR has a “pragmatic philosophical,” not positivist, underpinning (p. 6). Most professors are trained in a positivist 

research tradition. This view seems supported by Dede (2004), who notes that scholars trained in traditional research 

are likely to view a DBR approach as not very promising. 

As I have pointed out, rapid prototyping may be a useful design approach within higher education, and rapid 

prototyping as a design methodology is congruent with DBR; yet, little literature documents the experiences of 

professors as they prototype materials and processes. This paper begins filling this void in the literature. 

The Design Context and Events 

In this section, I provide an overview of one experience where I used rapid prototyping as a methodology for 

developing instructional strategies in the context of higher education. My purpose here is not to offer a full treatment 

of the processes and products of this design, as this information has been reported elsewhere (e.g., Knowlton, 2006; 

Knowlton 2004b; Knowlton 2004c). Rather, my purpose is to provide a context for the personal narrative that is 

offered in the next section of this paper. In this section, I describe the context in which my design activities occurred, 

and I describe the prototyping of the assignment across three phases of implementation. Within the discussion of 

each phase, I describe the ways that the evolving context influenced design decisions and how each phase was 

evaluated. 

Context in which Rapid Prototyping Design Occurred 

Barab and Squire (2004) note the importance of considering the broader context in which a design is developed and 

implemented. Furthermore, Collins, Joseph, and Bielaczyc (2004) note that “setting” is a critical variable for 

characterization when reporting DBR efforts. In this case, the macro context in which the design was implemented 

played a large factor in the progression of the designs and, more to the exact point of this paper, in the ways that I 

found myself situated within the design process. 

210

Rapid prototyping was used in the context of a two-year, field-based teacher-certification program that was designed 

to prepare undergraduate students for careers as public school teachers. These preservice teachers were assigned to 

K-12 classrooms in partnership schools. During the first year of the two-year program, the preservice teachers often 

served as paraprofessionals or aides. During the second year, though, the preservice teachers became more centrally 

involved in teaching and learning activity. 

Throughout the two years, a team of university faculty supervised weekly content seminars, which largely resembled 

“traditional” college classrooms. I was responsible for teaching educational psychology within the seminar. In 

principle, though, “courses” were non-existent. Instead, each courses’ content was integrated into seminar activities 

and discussions. Rapid prototyping was used to design strategies to support the effective use of computer-mediated 

bulletin board (CMBB) discussion. The rapid prototyping of strategies as a means of promoting learning is common 

(Stokes & Richey, 2000). 

Phases of Design and Implementation 

Table 1 provides an overview of the factors that influenced the initial, intermediate, and refined versions of design 

and the characteristics of each design. Each version corresponds to a school semester. Also shown in Table 1 is a 

summary of the evaluation findings that provided a source on which to base the subsequent iterations of design. 

Table 1. Overview of three prototypes of CMBB discussion guidelines 

Initial Version 

Intermediate Version Refined Version 

Factors Influencing Design Factors Influencing Redesign Factors Influencing Redesign 

• Need for flexible and efficient 

communication tool 

• Emerging nature of the field 

experience 

• Lack of information about the 

participants’ knowledge and 

skills 

• Need for basic content principles 

• Initial version was ineffectual 

• Shifting Responsibilities of 

preservice teachers 

• Changes to the use of seminar 

time 

• Evaluation of intermediate 

version 

• Elimination of seminar time for 

educational psychology 

Initial Design Characteristics: Characteristics of Design: Characteristics of Redesign: 

• Laissez-faire 

• Preservice teachers were simply 

made aware that discussion 

board existed. 

• Preservice Teachers assigned 

to two groups 

• Discussion based on threeweek 

cycles 

• Discussion centered on 

student-initiated problems and 

proposals for practical 

solutions 

• Addition of a Privacy 

Statement and job aid of CMC 

conventions 

• Additional direction to govern 

discussion contributions (e.g. 

focus on “instructional 

problems” only; more 

scaffolding of what constitutes 

a “good” contribution) 

• Addition of the self report 

form 

Evaluation of Initial Evaluation of Intermediate Evaluation of Refined 

• Ineffectual and rare use 

• Preservice teachers reported that 

they didn’t see practical value of 

using CMC 

• Problems were narrow in scope 

• Interaction among the 

preservice teachers was limited 

• Grading was cumbersome 

• Perservice Teachers noted 

workload was heavy and 

contrived 

• Reducing the number of required 

contributions was helpful in 

terms of the usability of the 

assignment 

• Scaffolding questions were 

useful in helping the preservice 

teachers think more broadly 

about applications of educational 

psychology 

211

Initial Design 

Because the field-based program was new, the context of the field experience emerged as implementation 

progressed. This symbiosis between context and implementation influenced the ways the bulletin board might be 

used. Beyond the newness of the field program, two other contextual factors influenced design: 

• I had no knowledge of the computer skills of the learner for which I was designing. Had they used a CMBB 

before? Did they even have skills to find the bulletin board and log on? 

• The preservice teachers had never before taken educational psychology. Certification tests that the preservice 

teachers would need to pass suggested the need for the preservice teachers to obtain a basic understanding of 

educational psychology concepts and principles. 

Because of these contextual factors, no specific strategies were designed to support the CMBB discussion. The 

faculty team simply made the preservice teachers aware that WebCT (the university’s course management tool) had 

a discussion board for asynchronous sharing of ideas. This laissez-faire approach resulted in virtually no use of the 

bulletin board. Some preservice teachers suggested that it was nice to know the bulletin board was available, but they 

did not see how sharing ideas on the bulletin board would help them prepare for working in their classrooms. 

Intermediate Design 

Beyond the obvious inadequacies of the initial use of the electronic bulletin board, a conflict between the emerging 

role of preservice teachers and a decision made by university personnel necessitated formalized guidelines to support 

the use of the bulletin board as an educational tool. The preservice teachers were moving from serving as 

paraprofessionals who assisted the teacher to professionals who were responsible for designing and implementing 

lesson plans. As they grew into these professional roles, they needed to experience a shift from knowing theory as 

described in textbooks to using theory as a basis of their problem-solving efforts. CMBBs are appropriate tools for 

supporting problem-solving within field experiences (Beckett & Grant, 2003). In spite of the preservice teachers 

becoming more authentically situated, the team of faculty members who supervised the weekly seminars decided that 

seminar time should be divided among content areas—“Today is an Educational Psychology seminar; next week will 

be a reading methods seminar.” Such a decision mitigates against the authenticity of a field experience. Designing 

strategies to support the use of the CMBB could facilitate continued integrated connections, even though seminar 

time was less integrated. 

I designed instructional strategies in the form of formal assignment guidelines that were similar to those already 

existing in the literature (cf., Knowlton, 2002). Participants were divided into two groups and the discussion was 

based on a three-week cycle of sharing and response. At the end of each cycle, roles were reversed so that preservice 

teachers in group one performed the responsibilities of the preservice teachers in group two and vice versa. 

During the first week of the discussion cycle, preservice teachers in group one were responsible for defining a 

professional problem that they were experiencing within their partnership school. During the second week of the 

discussion cycle, the preservice teachers in group two were responsible for using the textbook as a learning-ondemand 

resource to theoretically frame the problems that their colleagues had shared during week one of the cycle. 

During the third week of the discussion cycle, all the preservice teachers were responsible for three contributions to 

the bulletin board discussion. To build in reflection time for the preservice teachers, the assignment guidelines 

dictated that not all three contributions should be posted on the same day of the week. Because I wanted the 

preservice teachers focusing on dialogue, not earning a grade, I loosely structured assessment criteria, allowing them 

to receive most credit by participating in the discussion. 

My assessment of students’ efforts served as one basis for determining additional changes that could improve the 

efficacy of the CMBB assignment: “[O]nly the integration of assessment [with] evaluation can produce a clear 

picture of an online discussion’s educational viability” (Knowlton, 2001, p. 164). Through the synthesis of 

evaluation and assessment, several findings emerged: 

• The problems shared by the preservice teachers were extremely narrow in scope, with over 90% focusing on 

classroom discipline. 

• Most contributions during week three of the discussion were replies to the original problem posted during week 

one. In other words, the preservice teachers were not discussing the problems by interacting; they merely 

212

continued to offer solutions to the original problem. Furthermore, the ideas across solutions were highly 

redundant. 

• Grading overshadowed other activities that are related to assessment but more productive toward creating 

continued learning among students, such as my reacting to their discussion contributions, highlighting common 

themes among their interactions, and offering contributions to the discussion as an authentic participant. 

Certainly, grading is within the instructor’s purview, but it should not dominate assessment processes (Bauer & 

Anderson, 2000). 

Also, through formal written and oral feedback from the preservice teachers, I determined that numerous aspects of 

the discussion assignment should be modified: 

• The number of required contributions in both weeks two and three needed to be reduced. The preservice teachers 

indicated that the workload was simply too demanding. 

• Criteria that specified on what days of the week participation could occur needed to be eliminated. Several 

preservice teachers noted that they were printing out discussion contributions and sometimes even entire threads 

of discussion and reading them. So, while their actual contributions might come on a single day of the week, 

they were considering the discussion across time. 

• A “privacy policy” needed to be added. Some of the preservice teachers were concerned that the content of the 

online discussion might somehow “get back to” their mentor teachers, administrators, or even students and 

parents. Given the nature of some of the problems that were being shared, this could be embarrassing. 

Refined Design 

The evaluation of the intermediate implementation of the strategies contributed to the development of the refined 

design, but a change to the weekly seminars contributed, as well. It was determined that certain content areas— 

educational psychology being one such area—would not be given any formal emphasis during seminars. I was still 

accountable for assessing the preservice teachers and giving an Educational Psychology grade to each of them at 

semester’s end, yet I was afforded no formal seminar time to assess them. Continuing to use the CMBB seemed to be 

a choice that could help me overcome this dilemma. 

Three minor adjustments were made to the assignment guidelines in an effort to overcome some of the administrative 

problems discovered during the evaluation of the intermediate version. First, in the assignment guidelines I suggested 

the need to respect the privacy of all discussion participants by not sharing conversations from the bulletin board 

with others, such as school personnel. Second, in an effort to help the preservice teachers better consider the 

conventions of bulletin board discussion, I created a job aid that discussed some of these conventions, such as double 

spacing between paragraphs and using meaningful subject lines. Third, I developed a self-report form, which allowed 

the preservice teachers to report factual information to me about the frequency and scope of their participation. This 

form was not a self-assessment as much as it was a productivity report; it provided me with a list of threads in which 

I could find their contributions. This made the process of “grading” less time consuming. 

Beyond these administrative adjustments, though, three larger changes were made to the refined version of the 

assignment. 

• All problems contributed during week one of the discussion cycle must be “instructional problems”—as opposed 

to the type of behavior and discipline problems that dominated the intermediate version of the assignment. 

• The number of required contributions during week two of each cycle was reduced from three to two. 

• Week three contributions had to be replies to week two contributions, not replies to the original problem 

discussed during week one of each cycle. This change was designed to promote deeper analysis of the issues 

embedded within the problems, not just continued (and often redundant) “solutions” to the original problem. 

• A list of possible strategies that students might use as they offered a week three contribution was developed. See 

table 2 for a list of these strategies. 

The evaluation of the refined design was based on an open-ended survey completed by the preservice teachers. The 

survey was designed to capture the preservice teachers’ views of the strategies used to create participation. I focus 

here on feedback from the preservice teachers that directly relates to changes that I made in designing the refined 

version. The preservice teachers reported that 

213

• reducing the number of required contributions to the discussion was helpful in making the discussion more 

manageable. 

• focusing on instructional problems, as opposed to behavioral problems, was difficult, but the refocusing of their 

thinking did help them see broader applications of educational psychology. 

• the scaffolding questions shown in table 2 were somewhat useful in promoting discussion, but those same 

scaffolding questions seemed to limit the direction of the conversations too much. 

Table 2. Discussion prompts for week three contributions 

• Pick two replies to the same problem and discuss why you think one would work better than the other. 

• Pick a reply to a problem and discuss the strengths and weaknesses of the proposed solution 

• Pick a theory that someone mentioned as a help to understanding week #2 and apply that theory differently (or 

more thoroughly). 

• Discuss your experiences with how a solution has/has not worked in the classroom. 

• Write a summary of responses to your own problem and describe what the biggest things that you are taking 

away from your problem are. 

A Personal Account of Theory Development through Design 

Edelson (2002) has suggested that design research is different from design, and he points to the notion that “design 

research explicitly exploits the design process as an opportunity to advance the researchers (sic) understanding of 

teaching, learning, and educational systems” (p. 107). In the remainder of this paper, this exploitation comes in the 

form of a personal narrative—an exploitation of “the phenomenon of experience” (Clandinin & Connely, 2000, p. 

128). Indeed, the personal self does not operate apart from the professional self (Knowlton, 1995), and including a 

synthesis of the two can broaden the perspectives from which DBR is considered. In this respect, the remainder of 

this paper takes the “hermeneutical turn” (O'Grady, Righy, & Van den Hengel, 1987, p. 94), whereby I do not offer a 

definitive truth about design but an interpretation of my design experiences. Edelson notes that several types of 

theories can be created through participation in design research; this section is organized around each type of theory, 

with personal narrative integrated into each section. Admittedly, there is overlap among these three sections. Table 

three offers an overview of the coming discussion. 

Table 3. Learning through design within the context of the current project 

Domain Theories Design Frameworks Design Methodologies 

• The ill-structured nature of a • Balance between instructional • Structural revision is often 

field experience may hinder 

prescription and the natural 

necessary in order to bring a 

learning unless designers account benefits of social learning is design to full fruition 

for the lack of structure through 

difficult to achieve 

• Rapid prototyping allows for a 

their design decisions 

• Tension exists between a 

stronger co-dependence 

• A context for learning may not designer attempting to guide between theory and practice. 

influence desirable outcomes as 

learners through prescriptive 

much as the learner’s perceptions of processes and teaching those 

the context influences outcomes 

processes as a generalized 

cognitive tool. 

• Balance must be struck 

between the role of designers 

and role of facilitators 

Domain Theories 

Domain theories, Edelson (2002) notes, are “theor[ies] about the world, not [theories] about design per se.” One 

type of domain theory is the context theory, which deals with “the challenges and opportunities” presented by the 

context in which the educational intervention was applied (p. 113). Because my point is to consider the way that I 

encountered this design project as a situated phenomenon (i.e., design within context), I emphasize context theories; 

214

ut from these context theories, I make connections to outcome domain theories, which are both the “desired” and 

“undesirable” outcomes that are “associated with some intervention” (p. 113). Specifically, I examine the influence 

of the field-based context on my ability to promote the sound use of the CMBB. Then, I consider the CMBB itself as 

a context for problem solving. 

Field-based Context 

Field-based programs can be educationally valuable, but this was only the second implementation of this field-based 

certification program, and much of the context supporting the program was still developing. Administration included 

no clear plan—what Edelson (2002) calls a “sequence of design” (p. 114)—that would help fulfill the conceptual 

purpose of the field-based program. Furthermore, the various faculty members who were working in the field-based 

program each had individual visions of the program’s intended conceptual outcomes. Both the lack of administrative 

leadership and the absence of a shared vision among faculty members made it difficult to teach and facilitate learning 

in such a way as to support the very point of a field-based program—that content should be integrated and directly 

based on the preservice teachers’ field experiences (cf., Bell, 1995; Scanlon & Ford, 1998; Weber, 1996). My design 

activities within this nebulous field-based context lead me to a domain theory, and it can be stated as follows: In 

theory, the authenticity of a field-based context provides a unique environment that can allow learning to flourish in 

ways not possible in artificial environments; in practice, however, people may unknowingly undermine the benefits 

of being in an authentic context by attempting to bring a sense of familiar and artificial classroom structure to 

alleviate the dissonance created by the authentic and ill-structured field-based environment. 

In this case, it was not the learners who wanted the type of clear-cut processes and direct answers that would be 

indicative of an artificial environment. Rather, it was the faculty team that tried to bring too much structure—to 

enclose and box the phenomenon of experience. For learning to flourish within a field experience, designers must 

aim for an approach that accounts for the unpredictable and nebulous milieu, which would require them to accept the 

act of design as a complex phenomenon. Instead, this design team of which I was a part viewed the emerging context 

of the field-based program as a barrier to learning and imposed an artificial structure on learning processes. 

How did this artificial structure manifest itself? The faculty team regressed toward the expedient, familiar, and 

comfortable, not towards the educationally sound. Why did the faculty team, for example, keep changing the 

purpose, intent, and function of the weekly seminars? Was it in an effort to improve learning within the authentic 

context, or was it a type of “giving up” on the true benefits of a field-based context? I may seem to be arguing 

towards an indictment of the faculty team on which I served, but as I reflect on my own design decisions, I see a 

similar pattern of craving the familiarity of a traditional and familiar classroom environment and thus undermining 

the natural benefits of authenticity within a field-based context. What was my purpose in adding the self-report form 

in the refined design? Did such a form add to the preservice teachers’ educational experience? It did not, in my 

view. I was more concerned with administrative ease than with focusing on the preservice teachers’ learning. To put 

this in the language of an outcomes theory, the “sequence of design and implementation [of] cycles” (Edelson, 2002, 

p. 114) must be congruent with the learning context; without such congruence, “desirable outcomes” (p. 114) will 

not be realized. 

CMBB as Context for Problem Solving 

As I have noted, Edelson (2002) points to two types of domain theories, a context domain theory and an outcome 

domain theory. What is the relationship between a CMBB as context and the intended outcome of problem solving? 

Jonassen (2001) points out that different types of problems require different types of representations; thus a question 

about CMBB as an appropriate medium (i.e., context) for solving problems (i.e., the outcomes) is begged: Did the 

CMBB support representations that allowed the preservice teachers to move toward solutions? 

Contextually, a CMBB may seem to remove cues that contribute to a communicative environment. As Weiss (2000) 

notes, some of these cues—gestures, facial expressions, and other physical elements, for example—can “contribute 

subtle (and sometimes not so subtle) meanings or attitudes” (p. 48). So, an argument can be made that CMBB as a 

context for problem solving can, in fact, impede communication, and thus hinder the potential for learning through 

problem solving. During all phases of implementation, the preservice teachers did communicate to me informally 

215

that they felt this hindrance; I felt it, too. For example, recall that in the intermediate design I struggled to see a 

developing conversation. I observed static contributions to the discussion that offered solutions to the original 

problem without considering other contributions to the same thread of discussion. Yet, during the face-to-face 

seminars, conversations about the problems shared in the bulletin board were lively and highly interactive. 

Through my role in this project, however, I did come to see an opposing view of CMBBs as a context for problem 

solving. The absence of visual and audible cues, in fact, removed various elements that traditionally have lead to bias 

and disenfranchisement. Because a CMBB is a text-based environment, hidden were oral communication 

idiosyncrasies, such as regional accents, speech impediments, or a general lack of verbal eloquence. I, myself, came 

to this Midwestern university from my native Mississippi, and students do occasionally point to my distinct southern 

accent, which begrudgingly admitted influences my face-to-face interactions. As another example, one preservice 

teacher in particular had some interesting ideas but struggled mightily to formulate those ideas and articulate them 

when called upon to do so during the face-to-face seminars. His struggles manifested themselves in stream of 

conscious soliloquies that often seemed off topic; his body language seemed to indicate that he was aware of his 

inability to join the flow of the discussion. His struggles and physical manifestations of those struggles were clearly 

recognized by other preservice teachers many of whom hesitated to respond to his ideas during the seminar for fear 

of exacerbating his communication difficulties. This same preservice teacher, though, was better able to take 

advantage of the CMBB context and more carefully articulate his viewpoints in writing. As a result, this preservice 

teacher and others who responded to him were less “put on the spot” to immediately interact in productive ways. 

More substantively, race and sometimes gender were removed as influences that may have tainted the way the 

preservice teachers received messages from each other. The “pseudo-anonymity” of CMBBs (Kemp, 1998, p. 140) in 

some cases, then, promoted more comfortable interaction among the preservice teachers, which allowed the online 

experience to become more fully humanized. Paradoxically, by removing the familiarity of interacting through 

simplistic elements of voice, demeanor, and appearance, the preservice teachers’ very existence became inherently 

intertwined with their ideas. 

Above, I have described computer-mediated bulletin boards in terms of my realizations of them as a paradoxical 

context for communication. This context influences outcomes. These preservice teachers were in a context that was 

not familiar to them; thus the way that the context held potential for mediating their learning was uncomfortable for 

them. For positive outcomes to occur, then, the preservice teachers would need to see this paradox and come to shift 

their perceptions of the CMBB context. This point can be stated directly as a domain theory: Perhaps it is not a 

learning context as much as it is learners’ perceptions of (and comfort with) that context that determines whether 

desirable outcomes can be achieved. 

Particularly during the last semester of this project, the preservice teachers did informally note an understanding of 

this shift; though, as a practical matter, little evidence supported such an understanding. In terms of evidence, the 

resituating of the self as a result of the context may actually force a stronger awareness of sensory reactions. Facial 

expressions, for example, can only be represented as metalinguistic cues, such as emoticons. When traditional 

sensory cues that we often take for granted become abstract representations, participants must consciously search for 

opportunities to insert them into CMBB discussions. Admittedly, I did not see such a use of emoticons or other 

metalinguistics. This section does serve to suggest, though, a paradox regarding CMBBs as context. Furthermore, it 

offers a broad consideration of somewhat non-indigenous discussants within that context. 

Design Frameworks 

The domain-based considerations that were described in the previous section of this paper influenced my perceptions 

of both design frameworks, which are discussed in this section of the paper, and design methodologies, which are 

discussed in the next section of the paper. Edelson (2002) notes that a design framework is a “generalized design 

solution.” Design frameworks “describe the characteristics that a designed artifact must have [in order] to achieve a 

particular set of goals [with]in a particular context” (p. 114). The artifact was my articulation of the assignment 

guidelines to the extent that I shaped the assignment into a PBL and asynchronous discussion assignment. In this 

section, I discuss the shaping of the guidelines as both a problem-based learning artifact and an asynchronous 

discussion artifact. In considering both, I have come to learn that balance between two extremes is difficult to find. 

On one extreme is the careful design of strategies to promote learning; on the other extreme is the designer having a 

trust in students’ initiative and curiosity as a motivating impetus towards learning. 

216

Problem-based learning 

Designers must strike an important balance between allowing autonomy for students as problem-solvers and 

providing needed scaffolding to the same students, who can only be described as nascent in their problem-solving 

abilities. Within this project, a shift from complete autonomy (the initial design) towards strong scaffolding (the 

refined design) can be seen. At each extreme of this shift, I was taxed by questions about the educational viability of 

my design. As I embraced the laissez-faire approach of the initial design, I recognized that the preservice teachers 

may not know how to solve problems. Would they understand, for example, that articulating the problem is more 

important than offering solutions, a commonly-accepted principle of problem solving (cf., Abel, 2003)? Would they 

intuitively understand the inefficiency of rushing toward a solution without collaboratively exploring and analyzing 

alternatives based on their individual experiences (cf., Beckett & Grant, 2003)? Based on their use (or, better said, 

“the lack of use”) of the CMBB, I saw no evidence of positive answers to such questions—thus, the need for the 

intermediate design. 

As I began adjusting the assignment guidelines to better allow the preservice teachers to methodically solve 

problems, I felt that I perhaps was not doing justice to the potential of problem-based learning as a design 

framework. By articulating strategies, for example, was I not forcing a narrow view—my view—of how to articulate 

the problem space? Was I allowing the power of social interaction as a natural phenomenon to emerge, or was I 

creating something contrived that forced an artificial codependence among the preservice teachers? 

This tension can be framed more theoretically. Some constructivists claim that learning is “internally controlled and 

mediated by the learner” (Jonassen, 1991, p. 12). Yet, pedagogically, constructivists seem to focus on tools, 

environments, and interaction with others—all of which are external. To what extent within a PBL framework can a 

designer dictate the external requirements that will result in internal learning? Throughout this project, I felt 

dissonance regarding the balance between activity that was based on a teacher-centered design and activity motivated 

by the preservice teachers’ true cognitive dissonance that compelled them to pursue a consistent understanding of 

content. 

Have I set up a false dichotomy here by pointing to, on the one hand, a teacher-centered design and, on the other 

hand, student initiative? Perhaps I should have taught the preservice teachers a problem-solving methodology. 

Perhaps it was not an unwillingness to alleviate their own cognitive dissonance, as much as it was a lack of 

understanding about how to achieve this alleviation. As a design framework, problem solving does seem to be 

robust, but designers would do well to consider whether the robustness lies in guiding learners through the design 

process or teaching the problem-solving process as a generalized cognitive tool. 

Asynchronous discussion 

As one of my most prominent co-authors lives some five states away, I understand purposeful use of computer-based 

communication and the power that it has in representing ideas. Designing an artifact to help others see the power can 

be difficult, however. The power lies in one’s desire to engage in a collaborative dialogue; true “dialogical 

participation” is a higher-order type of learning through asynchronous discussion. It transcends the type of 

“generative participation” that can simply shape one’s individual thinking; dialogical participation moves 

asynchronous discussion participants toward principles of distributed learning (Knowlton, 2005). 

The initial design depended on the tool simply being available, which did not produce viable results. It is only when 

instructional strategies were added (in the intermediate design) and revised (in the refined design) that more 

participation and evidence of engagement began to appear. My experiences here seem to confirm Clark (1983, 

1994a, 1994b) and others’ assumption that computers do not create learning. Rather, it is the careful design of 

strategy that creates learning. 

There was a contradiction within asynchronous discussion as a design framework that I needed to reconcile. 

Designing strategies to “force” dialogue seemed necessary for promoting learning, yet I was committed to the idea 

that there must be some commitment on the part of the preservice teachers to want to engage in an academic 

discussion. Do the design of strategies mitigate against helping students see the need for dialogue? As I added the 

necessary characteristics that would allow asynchronous discussion as a framework to “achieve a particular set of 

217

goals” (Edelson, 2002, p. 114) was I not further usurping students’ authority? Consider the discussion prompts 

described earlier and shown in table 2. Does such a list of strategies send the message to the preservice teachers that 

week three contributions should involve close-ended and narrow responses, not attempts to contribute to an authentic 

conversation? 

I infer from the experiences that I had within this design project that one necessary characteristic of asynchronous 

discussion as a framework is an impetus to promote both initial contributions and replies. Thus, the designed artifact 

required both. The problem with these strategies is that they were discrete and perhaps aimed in the wrong direction. 

Perhaps appropriate design would have occurred more in my role as facilitator of a discussion, as opposed to 

designer of the guidelines. If I had facilitated the preservice teachers’ efforts to engage in dialogue, as opposed to 

“demanding” dialogue through the written assignment guidelines, then perhaps the preservice teachers would better 

have come to understand the benefits of peer-to-peer sharing of ideas. 

Design Methodologies 

Design methodologies refer to the “process for achieving a class of designs” (Edelson, 2002, p. 115). Edelson points 

to instructional systems design and human-computer interaction as examples of classes of design. Earlier in this 

paper, rapid prototyping was defined through existing literature, but the differences between encountering rapid 

prototyping through literature and personally experiencing it through an act of design are vast. Indeed, numerous 

variables specific to the project described in this paper have led to my understanding of rapid prototyping as a 

“system” of contextual inputs and student outputs. These inputs and outputs include contextual influences on my 

design decisions and the evaluation of each implementation as summarized in table 1. Also, though, my domain 

theories as they emerged throughout this project and the design frameworks described in the previous section of this 

paper shaped my understanding of rapid prototyping. If design—even the design of instructional strategies—is 

problem solving, then the representation of inputs, outputs, and change in designer thinking is consequential; as 

Jonassen (2003) notes, problem solving necessitates representation. This seems congruent with Edelson’s (2002) 

point that design can sometimes only be understood reflectively, after design has occurred. In the abstract, the 

system of inputs and outputs is shown in figure 1. 

Designer’s: 

• Domain Theories 

• Design Frameworks 

• Design Methodologies 

• Understanding of Project 

Variables 

Implementation 

Figure 1. Representation of rapid prototyping 

Usability 

Learning 

viability 

Practicality 

As figure 1 suggests, a designer’s existing domain theories, design frameworks, and understanding of design 

methodologies as well as the designer’s existing understanding of project variables may contribute to the prototyping 

and implementation of a design. Emerging from this implementation are at least three types of outputs: Usability, as 

defined by the learner; learning viability, as determined by a course instructor or evaluation personnel; and 

practicality given the constraints of a design scenario, as determined by administration or other stakeholders on the 

design team or within a context. Strikingly, only one of these three is directly concerned with learning, as neither 

usability nor practicality are necessarily related to learning processes. These outputs from the implementation are 

218

considered by a designer (or a design team) and become a type of input that shapes the designer’s continued thinking 

about theories, frameworks, methodologies, and project variables. 

When presented in this way, rapid prototyping can be seen as cyclical and can allow for infinite revisions across 

time. Such a representation begs two questions that should be explored to construct a fuller understanding of this 

conception of rapid prototyping methodology. The first relates to developing a true understanding of notions of 

“revision.” The second relates to the codependent interactions between theory and practice. 

What is the Nature of Revision? 

Design as process is not something to be tolerated; rather, it is something to be embraced. Largely absent from figure 

1 and the description is a more nuanced consideration of design’s subphases. I note, for example, that I drafted 

assignment guidelines in preparation for implementing the intermediate design. But, the creation of those assignment 

guidelines required multiple drafts characterized by structural revision. For example, in one draft of the intermediate 

guidelines, the preservice teachers were going to be divided into three groups of sharing problems and responding, 

not the two groups that I ultimately implemented. As I drafted a particular approach to a strategy, I felt tension 

among learner usability, practicality, and my primary directive of promoting learning among the preservice teachers. 

As a result of this tension, both the intermediate and refined drafts of the assignment guidelines were drafted into 

many different permutations. When a certain permutation didn’t “feel right” to me as the designer, I kept drafting to 

see where the design process would “take me.” This point may, in some ways, seem to be a statement of the 

obvious, but on average have those who engage in design based-research and curriculum development been trained 

in what it means to do more than “tolerate” the need for revision of a framework in an attempt to fully execute the 

methodology? Notions of “learning by design” as described by Edelson (2002) and others (e.g., Knowlton, 2004; 

Nelson, 2003; Nelson & Knowlton, 2005) require an understanding of documenting design efforts as a means of 

coming to understand the design that is attempting to emerge. 

How Do Theory and Practice Depend on Each Other? 

Through this project, I discovered a non-linear and co-dependent relationship between theory and practice within a 

rapid prototyping methodology. This non-linear relationship is particularly pronounced when rapid prototyping is 

compared with more traditional instructional design. In traditional instructional systems design, instructional theory 

guides and shapes design decisions (Morrison, Ross, & Kemp, 2004). The flow of thinking runs from instructional 

theory into a functional model (e.g., Morrison, Ross, & Kemp, 2004; Dick & Carey, 1990), which helps a designer 

make decisions. By the time implementation occurs a theoretical frame supporting practice has been solidified. In 

rapid prototyping, however, iterations and cycles of design and action occur more rapidly—the very essence of rapid 

prototyping. Thus, not only does theory shape design but also learner activity itself shapes future design decisions. 

Sometimes learner activity changes as a direct result of the implementation of a design. But, sometimes learner 

activity changes as the result of an evolution of the context in which learners are active. Regardless of the impetus 

for change, subsequent design decisions are affected by the change. I am not arguing that such a symbiotic 

relationship between theory and practice does not exist within traditional design; I merely am arguing that the 

relationship is more pronounced in rapid prototyping and designers need to be more aware of the relationship. 

Two examples from the current project might be illustrative. The first example illustrates changes as a result of a 

design decision, but the design decision was a response to learner activity. Consider the influence of strategies for 

enhancing social interaction that I added in the intermediate design. The design decisions to “force” interaction 

created a shift from individual cognition to a distributed view of cognition. This design decision was a direct 

response to the preservice teachers’ unwillingness (or inability) to collaborate without a sense of being “forced” to 

interact. This forced interaction, though, changed my thinking as a designer. Thus, in the refined design, I tweaked 

the interaction through additional design decisions. Namely, as described earlier, I placed criteria for success on their 

week three contributions to the discussion. 

A second example illustrates how design decisions can be a response to the ever-shifting context in which learners 

are engaged in learning activities. As I described earlier in this paper, as the field experience progressed, the 

preservice teachers became more authentically situated within the context of their classrooms; that is, they moved 

219

more and more from the periphery of classroom activity to the center of teaching and learning. In authentic cases, 

this is a natural progression that is justifiable (Lave & Wenger, 1991), but the point is that design decisions needed to 

be responsive to this shift. Specifying that problems worthy of discussion were to be instructional problems, not 

classroom discipline problems, was a design decision that I made in the refined version of the assignment. This 

design decision was purposeful toward the goal of broadening the preservice teachers’ thinking regarding what 

constitutes a classroom problem that is worthy of analysis. It was necessary to broaden the preservice teachers’ 

thinking because their shift in responsibilities was a broadening of the scope of things that fell under their purview. 

The strategies had to respond to the ever-evolving context in which the preservice teachers were engaged. 

Design decisions, then, serve the purpose not only of rectifying design problems through a reconsideration of 

theoretically-derived prescriptions but also of realigning learner practices within an ever-shifting context in which 

learning occurs and, as a direct result of that shifting context, the theoretical thinking of the designer. I have 

described some of these shifts in theoretical thinking—whether to err on the side of student initiative or to place 

more faith in strategies, for example—in the previous part of this paper. This conflict regarding on which side I 

should err was a direct result of observing my designs as they influenced the preservice teachers operating within an 

ever-evolving learning context. 

Implications 

In this paper, I have offered my experiences as a designer within a specific scenario. I have tried to offer this 

perspective from a “personal” viewpoint, but I have tried to shape the narrative provided here around what Edelson 

(2002) says a designer should “learn” (ie., what theories a designer should develop) as design is occurring. The 

narrative, in its own right, provides insights about DBR as a mode of inquiry and insights about notions of rapid 

prototyping. In one respect, I am simply pointing to a practical operationalization of Edelson’s (2002) view that the 

problem or possibility that leads to design often develops hand-in-hand with the design itself. In a larger respect, I 

am pointing to the idea that there is value, simply, in designers coming to understand the multi-faceted influences on 

design decisions. In what follows, I provide heuristical questions for would-be designers. The heuristical questions 

might be useful in the confines of faculty development initiatives in which faculty are engaged in design (e.g., 

Nelson & Knowlton, 2005). Through these questions, I am not suggesting the need for quantitative empiricism; 

rather, I am advocating a deeper excursion into hermeneutic approaches for understanding DBR. 

What is the relationship between designer training and perceptions of a design scenario? 

I am trained in the neo-classical tradition of instructional systems design. Would someone trained in a different 

tradition—say, in user-engineering or constructivist environment design—have seen this design scenario from a 

different perspective? Certainly, they would have, but research needs to be done in how a designer’s formal or 

experiential training influences that designer’s understanding of a design scenario. Designers who report their efforts 

through personal narratives should share their biases and assumptions about design in order to unveil the perspective 

from which the designer is coming. 

How does context itself influence designer’s perceptions? 

Like many scenarios described in the literature, I was operating here in the context of teacher-education. What if my 

experiences were based in a similar task (i.e., designing strategies to support the educational use of CMBB) but in a 

different context (e.g., in corporate management training)? Certainly there are cultural differences among teacher 

education, corporate education, and other contexts in which formal learning activities must be designed. How do 

those macro-context cultures influence designers’ perceptions of their tasks? Consider, for example, the stark 

differences between a designer in a corporate setting and a college professor acting as designer. Likely, the corporate 

designer will not also be facilitating the implementation of design. Yet, in the context of higher education, the 

designer is often the same person who implements the designs. Designers in a variety of settings should offer their 

personal perceptions of design tasks and scenarios. Through these perceptions, a variety of contexts for personal 

narratives would be present in the literature. 

220

How would one of the Preservice Teacher’s Narratives Align with the Narrative Presented here? 

Often in considering a learning scenario, the perceptions of the professor (acting as both designer and facilitator of 

learning) may be markedly different from the perceptions of students who experienced the learning side of design 

(cf., Knowlton, Eschmann, Fish, Heffren, & Voss, 2004). Admittedly, this paper has focused on the hermeneutical 

perspective of a single individual. The perspective provided by personal narrative is powerful, but admittedly, 

accompanying perspectives would add research robustness. It would be interesting to see a parallel discussion from 

one of the preservice teachers involved in this project. Designers might consider structuring their personal narratives 

and setting them against a frame of learner narratives. Such an approach would better allow multiple data sources 

(albeit still ones based in hermeneutics and personal narrative) to be considered. 

References 

Abel, C. F. (2003). Heuristics and problem solving. In D. S. Knowlton & D. C. Sharp (Eds.), Problem-based 

learning in the information age, San Francisco: Jossey-Bass, 53-58. 

Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning 

Sciences, 13 (1), 1-14. 

Bauer, J. F., & Anderson, R. S. (2000) Evaluating students’ written performance in the online classroom. In R. E. 

Weiss, D. S. Knowlton & B. W. Speck, (Eds.), Principles of effective teaching in the online classroom, San 

Francisco: Jossey-Bass, 65-72. 

Beckett, J., & Grant, N. K. (2003). Guiding students toward solutions in field experiences. In D. S. Knowlton & D. 

C. Sharp (Eds.), Problem-based learning in the information age, San Francisco: Jossey-Bass, 67-72. 

Bell, N. (1995). Professional development sites: Creating authentic experiences for preservice middle school 

teachers. Current Issues in Middle Level Education, 4 (1), 9-19. 

Clark, R. E. (1983). Reconsidering research on learning from media. Review of Educational Research, 53 (4), 445- 

459. 

Clark, R. E. (1994a). Media will never influence learning. Educational Technology Research and Development, 42 

(2), 21-29. 

Clark, R. E. (1994b). Media and method. Educational Technology Research and Development, 42 (3), 7-10. 

Clandinin, D. J., & Connelly, F. M. (2000). Narrative inquiry: Experience and story in qualitative research, San 

Francisco: Jossey-Bass. 

Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design research: Theoretical and methodological issues. The 

Journal of the Learning Sciences, 13 (1), 15-42. 

Dede, C. (2004). If design-based research is the answer, what is the question? A commentary on Collins, Joseph, and 

Bielaczyc; diSessa and Cobb; and Fishman, Marx, Blumenthal, Krajcik, and Soloway in the JLS special issue on 

Design-based research. The Journal of the Learning Sciences, 13 (1), 105-114. 

Dick, W., & Carey, L. (1990). The systematic design of instruction (3 rd Ed.), New York: Harper Collins. 

diSessa, A. A., & Cobb, P. (2004). Ontological innovations and the role of theory in design experiments. The Journal 

of the Learning Sciences, 13 (1), 77-103 

Edelson, D. C. (2002). Design research: What we learn when we engage in design. The Journal of the Learning 

Sciences, 11 (1), 105-121. 

221

Edelson, M. (1988). The hermeneutic turn and the single case study in psychoanalysis. In D. N. Berg & K. K. Smith 

(Eds.), The self in social inquiry: Researching methods, Newbury Park: Sage, 21-34. 

Jonassen, D. H. (2001). Toward a design theory of problem solving. Educational Technology Research & 

Development, 48 (4), 63-86. 

Jonassen, D. H. (2003). Using cognitive tools to represent problems. Journal of Research on Technology in 

Education, 35 (3), 362-381. 

Jonassen, D. H. (1991). Objectivism versus constructivism: Do we need a new philosophical paradigm? Journal of 


Kemp, F. (1998). Computer-mediated communication: Making nets work for writing instruction. In J. R. Galin & J. 

Latchaw (Eds.), The dialogic classroom: Teachers integrating computer technology, pedagogy, and research, 

Urbana: National Council of Teachers of English, 133-150. 

Knowlton, D. S. (2006). Rapid prototyping as method for developing instructional strategies for supporting 

computer-mediated communication among university students. The Journal for the Scholarship of Teaching & 

Learning, 6 (1), 75-87. 

Knowlton, D. S. (2005). A taxonomy of learning through asynchronous discussion. Journal of Interactive Learning 

Research, 16 (2), 155-177. 

Knowlton, D. S. (2004a). Never mind the prescriptions, bring on the descriptions: Students’ representations of 

inquiry-driven design. In M. Simonson & M. Crawford (Eds.), The Proceedings of the 27 th Annual Convention of the 

Association for Educational Communications and Technology, Bloomington, IN: AECT, 369-374. 

Knowlton, D. S. (2004b). Electronic bulletin boards as medium for asynchronous problem solving in field 

experiences. The International Journal of Instructional Technology & Distance Education, 1 (5), 43-52. 

Knowlton, D. S. (2004c). Using asynchronous discussion to promote collaborative problem solving among 

preservice teachers in field experiences: Lessons learned from implementation. In M. Simonson & M. Crawford 

(Eds.), The Proceedings of the 27 th Annual Convention of the Association for Educational Communications and 

Technology, Bloomington, IN: AECT, 375-381. 

Knowlton, D. S. (2002). Promoting liberal arts thinking through online discussion: A practical application and its 

theoretical basis. Educational Technology & Society, 5 (3), 189-194. 

Knowlton, D. S. (2001). Determining educational viability in online discussions: A student-centered approach. 

Academic Exchange Quarterly, 5 (4), 162-168. 

Knowlton, D. S. (1995, November). Personal narrative and graduate-level education: How does gender influence 

thinking and writing about curriculum? Paper presented at the annual meeting of the Mid-South Educational 

Research Association, November 8-10, 1995, Biloxi, Mississippi. 

Knowlton, D. S., Eschmann, A., Fish, N., Heffren, B., & Voss, H. (2004). Processes and impact of journal writing in 

a graduate-level theory course: Students’ experiences and reflections. Teaching & Learning: The Journal of Natural 

Inquiry and Reflective Practice, 18 (2), 40-53. 

Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation, Cambridge, MA: Cambridge 


Lohr, L., Javeri, M., Mahoney, C., Gall, J., Li, K., & Strongin, D. (2003). Using rapid application development to 

improve usability of a preservice teacher technology course. Educational Technology Research & Development, 51 

(2), 41-55. 

222

Marshall, C., & Rossman, G. (1995). Designing qualitative research, Newbury Park, CA: Sage. 

Morrison, G. R., Ross, S. M., & Kemp, J. E. (2004). Designing effective instruction (4 th Ed.), New York: John Wiley. 

Nelson, W. (2003). Learning by design. In D. S. Knowlton & D. C. Sharp (Eds.), Problem-based learning in the 

information age, San Francisco: Jossey-Bass, 39-44. 

Nelson, W., & Knowlton, D. S. (2005). “Learning through Design” as a strategy for faculty development: Lessons 

learned in a teacher education setting. In M. Orey, J. McClendon, & R. M. Branch (Eds.), Educational media and 

technology yearbook (vol. 30), Westport: Libraries Unlimited, 29-36. 

O'Grady, P. J., Righy, P., & Van den Hengel, J. W. (1987). Hermeneutics and the method of social science. 

American Psychologist, 42 (2), 194. 

Patton, M. Q. (1990). Qualitative evaluations and research methods (Rev. Ed.), Newbury Park, CA: Sage. 

Reiser, R. A. (2001). A history of instructional design and technology: Part 1: A history of instructional media. 

Educational Technology Research and Development, 49 (1), 53-64. 

Scanlon, P. A., & Ford, M. P. (1998). Grading student performance in real-world settings. In R. S. Anderson & B. 

W. Speck (Eds.). Changing the way we grade student performance: Classroom assessment and the new learning 

paradigm, San Francisco: Jossey-Bass, 97-105. 

Stokes, T., & Richey, R. C. (2000). Rapid prototyping methodology in action: A developmental study. Educational 

Technology Research and Development, 48 (2), 63-80. 

Weber, A. (1996). Professional development schools and university laboratory schools: Is there a difference? The 

Professional Educator, 18 (2), 59-65. 

Weiss, R. E. (2000). Humanizing the online classroom. In R. E. Weiss, D. S. Knowlton, & B. W. Speck (Eds.), 

Principles of effective teaching in the online classroom, San Francisco: Jossey-Bass, 47-52. 

Wiggins, G., & McTighe, J. (2005). Understanding by design (2 nd Ed.). Alexandria: Association for Supervision and 

Curriculum Development. 

223

Wang, H.-Y., & Chen, S. M. (2007). Artificial Intelligence Approach to Evaluate Students’ Answerscripts Based on the Similarity 

Measure between Vague Sets. Educational Technology & Society, 10 (4), 224-241. 

Artificial Intelligence Approach to Evaluate Students’ Answerscripts Based on 

the Similarity Measure between Vague Sets 

Hui-Yu Wang 

Department of Education, National Chengchi University, Taiwan // 94152514@@nccu.edu.tw 

Shyi-Ming Chen 

Department of Computer Science and Information Engineering, National Taiwan University of Science and 

Technology, Taiwan // Tel: +886-2-27376417 // smchen@mail.ntust.edu.tw 

ABSTRACT 

In this paper, we present two new methods for evaluating students’ answerscripts based on the similarity 

measure between vague sets. The vague marks awarded to the answers in the students’ answerscripts are 

represented by vague sets, where each element ui in the universe of discourse U belonging to a vague set is 

represented by a vague value. The grade of membership of u i in the vague set Ã is bounded by a subinterval 

[tÃ(u i), 1 – f Ã (u i)] of [0, 1]. It indicates that the exact grade of membership μ Ã(u i) of u i belonging the vague set 

Ã is bounded by t Ã(u i) ≤ μ Ã(u i) ≤ 1 – f Ã(u i), where t Ã(u i) is a lower bound of the grade of membership of u i 

derived from the evidence for ui, f Ã(u i) is a lower bound of the negation of u i derived from the evidence against 

u i, t Ã(u i) + f Ã(u i) ≤ 1, and u i∈U. An index of optimism λ determined by the evaluator is used to indicate the 

degree of optimism of the evaluator, where λ ∈ [0, 1]. Because the proposed methods use vague sets to 

evaluate students’ answerscripts rather than fuzzy sets, they can evaluate students’ answerscripts in a more 

flexible and more intelligent manner. Especially, they are particularly useful when the assessment involves 

subjective evaluation. The proposed methods can evaluate students’ answerscripts more stable than Biswas’s 

methods (1995). 

Keywords 

Similarity functions, Students’ answerscripts, Vague grade sheets, Vague membership values, Vague sets, Index of 

optimism 

Introduction 

In recent years, some methods have been presented for students’ evaluation (Biswas, 1995; Chang & Sun, 1993; 

Chen & Lee, 1999; Cheng & Yang, 1998; Chiang and Lin, 1994; Frair, 1995; Echauz & Vachtsevanos, 1995; 

Hwang, Lin, & Lin, 2006; Kaburlasos, Marinagi, & Tsoukalas, 2004; Law, 1996; Ma & Zhou, 2000; Liu, 2005; 

McMartin, Mckenna, & Youssefi, 2000; Nykanen, 2006; Pears, Daniels, Berglund, & Erickson, 2001; Wang & Chen 

2006a; Wang & Chen, 2006b; Wang & Chen, 2006c; Wang & Chen, 2006d; Weon & Kim, 2001; Wu, 2003). Chang 

and Sun (1993) presented a method for fuzzy assessment of learning performance of junior high school students. 

Chen and Lee (1999) presented two methods for evaluating students’ answerscripts using fuzzy sets. Cheng and 

Yang (1998) presented a method for using fuzzy sets in education grading systems. Chiang and Lin (1994) presented 

a method for applying the fuzzy set theory to teaching assessment. Frair (1995) presented a method for student peer 

evaluations using the analytic hierarchy process method. Echauz and Vachtsevanos (1995) presented a fuzzy grading 

system to translate a set of scores into letter grades. Hwang, Lin and Lin, (2006) presented an approach for test-sheet 

composition with large-scale item banks. Kaburlasos, Marinagi, and Tsoukalas (2004) presented a software tool, 

called PARES, for computer-based testing and evaluation used in the Greek higher education system. Law (1996) 

presented a method for applying fuzzy numbers in education grading systems. Liu (2005) presented a method for 

using mutual information for adaptive item comparison and student assessment. Ma and Zhou (2000) presented a 

fuzzy set approach for the assessment of student-centered learning. McMartin, Mckenna and Youssefi (2000) used 

scenario assignments as assessment tools for undergraduate engineering education. Nykanen (2006) presented 

inducing fuzzy models for student classification. Pears, Daniels, Berglund, and Erickson (2001) presented a method 

for student evaluation in an international collaborative project course. Wang and Chen (2006a) presented two 

methods for students’ answerscripts evaluations using fuzzy sets. Wang and Chen (2006b) presented two methods 

for evaluating students’ answerscripts using fuzzy numbers associated with degrees of confidence. Wang and Chen 

(2006c) presented two methods for students’ answerscripts evaluation using vague sets. Weon and Kim (2001) 

presented a leaning achievement evaluation strategy in student’s learning procedure using fuzzy membership 






224

functions. Wu (2003) presented a method for applying the fuzzy set theory and the item response theory to evaluate 

learning performance. 

Biswas (1995) pointed out that the chief aim of education institutions is to provide students with the evaluation 

reports regarding their test/examination as sufficient as possible and with the unavoidable error as small as possible. 

Therefore, Biswas (1995) presented a fuzzy evaluation method (fem) for applying fuzzy sets in students’ 

answerscripts evaluation. He also generalized the fuzzy evaluation method to propose a generalized fuzzy evaluation 

method (gfem) for students’ answerscripts evaluation. In (Biswas, 1995), the fuzzy marks awarded to answers in the 

students’ answerscripts are represented by fuzzy sets (Zadeh, 1965). In a fuzzy set, the grade of membership of an 

element ui in the universe of discourse U belonging to a fuzzy set is represented by a real value between zero and 

one, However, Gau and Buehrer (1993) pointed out that this single value between zero and one combines the 

evidence for ui∈U and the evidence against ui∈U. They pointed out that it does not indicate the evidence for ui∈U 

and the evidence against ui ∈U, respectively, and it does not indicate how much there is of each. Gau and Buehrer 

(1993) also pointed out that the single value between zero and one tells us nothing about its accuracy. Thus, they 

proposed the theory of vague sets, where each element in the universe of discourse belonging to a vague set is 

represented by a vague value. Therefore, if we can allow the marks awarded to the questions of the students’ 

answerscripts to be represented by vague sets, then there is room for more flexibility. 

In this paper, we present two new methods for evaluating students’ answerscripts based on the similarity measure 

between vague sets. The vague marks awarded to the answers in the students’ answerscripts are represented by vague 

sets, where each element belonging to a vague set is represented by a vague value. An index of optimismλ(Cheng 

and Yang, 1998) determined by the evaluator is used to indicate the degree of optimism of the evaluator, 

whereλ∈[0, 1]. If 0 ≤λ< 0.5, then the evaluator is a pessimistic evaluator. Ifλ= 0.5, then the evaluator is a normal 

evaluator. If 0.5

If the universe of discourse U is an infinite set, then a vague set Ã of the universe of discourse can be represented as 

Ã = [ ~ ( ), 1 ~ ( ) ] / i , 

∫ t u 

U A i − f u 

A i u ui∈U, (2) 

where the symbol ∫ denotes the union operator. 

Figure 1. A vague set 

Definition 1: Let Ã be a vague set of the universe of discourse U with the truth-membership function tÃ and the falsemembership 

function fÃ, respectively. The vague set Ã is convex if and only if for all u1, u2 in U, 

whereλ ∈ [0, 1]. 

tÃ(U), 1- fÃ(U) 

1.0 

1 - ƒ Ã(u i) tÃ(U) 

t Ã(u i) 

0 

ui 

1- fÃ(U) 

tÃ(λ u1 + (1 –λ )u2) ≥ Min(tÃ(u1), tÃ(u2)), (3) 

1 – fÃ (λ u1 + (1 –λ ) u2) ≥ Min(1 – fÃ(u1), 1 – fÃ(u2)), (4) 

Definition 2: A vague set Ã of the universe of discourse U is called a normal vague set if ∃ ui ∈ U, such that 1 – 

fÃ(ui) = 1. That is, fÃ(ui) = 0. 

Definition 3: A vague number is a vague subset in the universe of discourse U that is both convex and normal. 

Chen (1995b) presented a similarity measure between vague values. Let X = [tx, 1 – fx] be a vague value, where 

tx∈[0, 1], fx∈[0, 1] and tx + fx ≤ 1. The score of the vague value X can be evaluated by the score function S shown as 

follows: 

S(X) = tx – fx, (5) 

where S(X)∈[-1, 1]. Let X and Y be two vague values, where X = [tx, 1 – fx], Y = [ty, 1 – fy], tx∈[0, 1], fx∈[0, 1], tx + 

fx ≤ 1, ty∈[0, 1], fy∈[0, 1], and ty + fy ≤ 1. The degree of similarity M(X, Y) between the vague values X and Y can be 

evaluated by the function M, 

S( 

X ) − S( 

Y ) 

M ( X , Y ) = 1− 

, (6) 

2 

where S(X) = tx – fx and S(Y) = ty – fy. The larger the value of M(X, Y), the higher the degree of similarity between the 

U 

226

vague values X and Y. It is obvious that if X and Y are identical vague values (i.e., X = Y), then S(X) = S(Y). By 

applying Eq. (6), we can see that M(X, Y) = 1, i.e., the degree of similarity between the vague values X and Y is equal 

to 1. 

Table 1 shows some examples of the degree of similarity M(X, Y) between X and Y. 

Table 1. Some examples of the degree of similarity M(X, Y) between the vague values X and Y 

X Y M(X, Y) 

[1, 1] [0, 0] 0 

[1, 1] [1, 0] 

1 

2 

[1, 0] [1, 1] 

1 

2 

[0, 1] [0, 1] 1 

Let X and Y be two vague values, where X = [tx, 1 – fx], Y = [ty, 1 – fy], tx∈[0, 1], fx∈[0, 1], tx + fx ≤ 1, ty∈[0, 1], 

fy∈[0, 1], and ty + fy ≤ 1. The proposed similarity measure between vague values has the following properties: 

Property 1: Two vague values X and Y are identical if and only if M(X, Y) = 1. 

Proof: 

(i) If X and Y are identical, then tx = ty and 1 – fx = 1 – fy (i.e., fx = fy). Because S(X) = tx – fx and S(Y) = ty – fy = tx – fx, 

the degree of similarity between the vague values X and Y is calculated as follows: 

S( 

X ) − S( 

Y ) 

M ( X , Y ) = 1− 

2 

( t x − f x ) − ( t y − f y ) 

= 1− 

2 

= 

= 1. 

(ii) If M(X, Y) = 1, then 

( t x − f x ) − ( t x − f 

1− 

2 

S( 

X ) − S( 

Y ) 

M ( X , Y ) = 1− 

2 

( t x − f x ) − ( t y − f y ) 

= 1− 

2 

x 

) 

= 1. 

It implies that tx = fx and ty = fy (i.e., 1 –ty = 1 – fy). Therefore, the vague values X and Y are identical. Q. E. D. 

Property 2: M(X, Y) = M(Y, X). 

Proof: 

Because 

and 

S( 

X ) − S( 

Y ) 

M ( X , Y ) = 1− 

, 

2 

S( 

Y ) − S( 

X ) 

M ( X , Y ) = 1− 

, 

2 

227

we can see that 

S( X ) − S( 

Y ) S( 

Y ) − S( 

X ) 

= 

, 

2 

2 

S( X ) − S( 

Y ) S( 

Y ) − S( 

Y ) 

1− 

= 1− 

2 

2 

and M(X, Y) = M(Y, X). Q. E. D. 

Let Ã and B ~ be two vague sets in the universe of discourse U, U = {u1, u2, …, un}, where 

Ã = [ t ~ ( u ), 1− 

f ~ ( u )] / u1 + [ t ( ), 1 ( )] 

A 1 A 1 

~ u − f 2 ~ u / u2 + … + [ t ( ), 1 ( )] 

A 

A 2 

~ un 

− f ~ u 

A 

A 

n / un, 

and 

B ~ = t ( u ), 1− 

f ( u )] / u1 + t ( u ), 1− 

f ( u )] / u2 + … + t ( u ), 1− 

f ( u )] / un. 

[ ~ 

B 1 ~ 

B 

1 

[ ~ 2 ~ 

B 

B 

2 

[ ~ 

B 

n ~ n 

B 

Let ~ ( i) 

A u V = [ ) ( ~ 

A i u t , 1 – ) ( ~ 

A i u f ] be the vague membership value of ui in the vague set Ã, and let ~ ( ) 

B i u V = 

[ ~ ( ) 

B i u t , 1 – ) ( ~ 

B 

i u f ] be the vague membership value of ui in the vague set B ~ . By applying Eq. (5), we can see 

that the score ( ~ ( )) 

A i u V S of the vague membership value ) ( ~ 

A i u V can be evaluated as follows: 

( ~ ( )) 

A i u V S = ) ( ~ 

A i u t – ) ( ~ 

A i u f , 

and the score ( )) V S of the vague membership value ) ( V can be evaluated as follows: 

( ~ 

B i u 

( )) V S = ) ( t – ), ( f 

( ~ 

B i u 

~ 

B i u 

~ 

B i u 

~ 

B i u 

where 1 ≤ i ≤ n. Then, the degree of similarity H(Ã, B ~ ) between the vague sets Ã and B ~ can be evaluated by the 

function H, 

n ~ ~ 1 

H ( A, 

B) 

= ∑ M ( V ~ ( u ), V u i ~ ( )) 

B i 

n i= 

1 

A 

1 ( ~ ( )) ( ~ ( )) 

∑ 1 

, 

= 1 

2 ⎟ ⎟ 

n ⎛ S V u − S V u ⎞ 

= ⎜ A i B i 

− 

(7) 

n i ⎜ 

⎝ 

⎠ 

where H(Ã, B)∈[0, 1]. The larger the value of H(Ã, B ~ ), the higher the similarity between the vague sets Ã and B ~ . 

Let Ã and B ~ be two vague sets in the universe of discourse U, U = {u1, u2, …, un}, where 

Ã = [ t ~ ( u ), 1− 

f ~ ( u )] / u1 + [ t ( ), 1 ( )] 

A 1 A 1 

~ u − f 2 ~ u / u2 + … + [ t ( ), 1 ( )] 

A 

A 2 

~ un 

− f ~ u 

A 

A 

n / un, 

and 

B ~ = [ t ~ ( u ), 1− 

f ~ ( u )] / u1 + [ t ( ), 1 ( )] 

B 1 

B 

1 

~ u − f 2 ~ u / u2 + … + [ ( ), 1 ( )] 

B 

B 

2 

~ un 

f 

B 

~ un 

B 

The proposed similarity measure between vague sets has the following properties: 

Property 3: Two vague sets Ã and B ~ are identical if and only if H(Ã, B ~ ) = 1. 

Proof: 

(i) If Ã and B ~ are identical, then 

t − / un. 

228

t ( u ), 1− 

f ( u )] = t ( u ), 1− 

f ( u )], where 1 ≤ i ≤ n. 

[ ~ ~ 

A i A i 

[ ~ 

B i ~ 

B 

i 

( ~ A i u ( ~ 

B i u 

That is, t ( ) = ) ( t , f ) = f ), and 1 ≤ i ≤ n. Because 

and 

~ 

A i u 

~ 

B i u 

( )) V S = ) ( 

( ~ 

A i u 

t – ) ( f 

~ 

A i u 

~ A i u 

( ~ ( )) 

B i u V S = ) ( ~ 

B i u t – ) ( ~ B i u f = ) ( ~ 

A i u t – ) ( ~ A i u f = )). ( ( ~ A i u V S 

Therefore, we can see that 

H(Ã, B ~ ) ∑ ⎟ = ⎟ 

n 1 ⎛ S( 

V ~ ( ui 

)) − S( 

V ~ ( ui 

)) ⎞ 

= ⎜ A 

B 

1− 

n ⎜ i 1 ⎝ 

2 

⎠ 

(ii) If H(Ã, B ~ ) = 1, then 

= ∑ ⎟ = ⎟ 

n 1 ⎛ S( 

V ~ ( ui 

)) − S( 

V ~ ( ui 

)) ⎞ 

⎜ A 

A 

1− 

n ⎜ i 1 ⎝ 

2 

⎠ 

= 1. 

H(Ã, B ~ ) ∑ ⎟ = ⎟ 

n 1 ⎛ S( 

V ~ ( ui 

)) − S( 

V ~ ( ui 

)) ⎞ 

= ⎜ A 

B 

1− 

= 1. 

n ⎜ i 1 ⎝ 

2 

⎠ 

It implies that ( )) V S = )), ( V S where 1 ≤ i ≤ n. Because )) ( V S = 

( ~ 

A i u 

( ( ~ 

B i u V ( ~ 

B i u 

( ~ 

B i u ( ~ A i u 

( ~ B i u 

( ~ B i u 

( )) V S and S )) = t ) – f ), where 1 ≤ i ≤ n, we can see that 

( ~ 

B i u 

t ( ) = ) 

~ 

A i u 

t and f ) = ) ( f (i.e., 1 – ) ( f = 1 – ) ( f ), 

~ B i u 

~ A i u 

~ B i u 

where 1 ≤ i ≤ n. Therefore, the vague sets Ã and B ~ are identical. Q. E. D. 

Property 4: H(Ã, B ~ ) = H( B ~ , Ã). 

Proof: 

Because 

H(Ã, B ~ ) ∑ ⎟ = ⎟ 

n 1 ⎛ S( 

V ~ ( ui 

)) − S( 

V ~ ( ui 

)) ⎞ 

= ⎜ A 

B 

1− 

n ⎜ i 1 ⎝ 

2 

⎠ 

and 

H( B ~ , Ã) ∑ ⎟ = ⎟ 

n 1 ⎛ S( 

V ~ ( ui 

) − S( 

V ~ ( ui 

)) ⎞ 

= ⎜ B 

A 

1− 

, 

n ⎜ i 1 ⎝ 

2 

⎠ 

and because 

∑ 

= 

⎟ ⎟⎟ 

⎛ ⎞ 

1 n S( 

V~ 

( u − 

⎜ 

i)) 

S( 

V~ 

( ui 

)) 

⎜ 

1− 

A B = ∑ n i 1 

2 

⎝ 

⎠ 

= ⎟ ⎟⎟ 

⎛ ⎞ 

1 n S( 

V~ 

( u − 

⎜ 

i) 

S( 

V~ 

( ui 

)) 

⎜ 

1− 

B A , 

n i 1 

2 

⎝ 

⎠ 

we can see that H(Ã, B ~ ) = H( B ~ , Ã). Q. E. D. 

( ~ A i u 

229

Example 1: Let Ã and B ~ be two vague sets of the universe of discourse U, 

U = {u1, u2, u3, u4, u5}, 

Ã = [0.2, 0.4]/u1 + [0.3, 0.5]/u2 + [0.5, 0.7]/u3 + [0.7, 0.9]/u4 + [0.8, 1]/u5 

B ~ = [0.3, 0.5]/u1 + [0.4, 0.6]/u2 + [0.6, 0.8]/u3 + [0.7, 0.9]/u4 + [0.8, 1]/u5, 

where 

V A 

~ ( u1 

V ~ ( u 

A 2 

V ~ ( u 

A 3 

V ~ ( u 

A 4 

) 

) 

) 

) 

V ~ ( u 

B 1 

V ~ ( u 

B 2 

V ~ ( u 

B 3 

V ~ ( u 

B 4 

V ~ ( u 

B 5 

= [0.2, 0.4], ) = [0.3, 0.5], 

= [0.3, 0.5], ) = [0.4, 0.6], 

= [0.5, 0.7], ) = [0.6, 0.8], 

= [0.7, 0.9], ) = [0.7, 0.9], 

V ~ ( u ) = [0.8, 1], ) = [0.8, 1]. 

A 5 

By applying Eq. (5), we can get 

= 0.2 – 0.6 = –0.4, 

S( 

V ~ ( u )) 

A 1 

S V ( )) 

( ~ u 

A 2 

S( 

V ~ ( u 

A 3 

S( 

V ~ ( u 

A 4 

S( 

V ~ ( u 

A 5 

S( V ~ ( u 

B 1 

S( V ~ ( u 

B 2 

S( V ~ ( u 

B 3 

S( V ~ ( u 

B 4 

S( V ~ ( u 

B 5 

)) 

)) 

)) 

)) 

)) 

)) 

)) 

)) 

= 0.3 – 0.5 = –0.2, 

= 0.5 – 0.3 = 0.2, 

= 0.7 – 0.1 = 0.6, 

= 0.8 – 0 = 0.8, 

= 0.3 – 0.5 = –0.2, 

= 0.4 – 0.4 = 0, 

= 0.6 – 0.2 = 0.4, 

= 0.7 – 0.1 = 0.6, 

= 0.8 – 0 = 0.8. 

By applying Eq. (7), the degree of similarity H(Ã, B ~ ) between the vague sets Ã and B ~ can be evaluated, shown as 

follows: 

H(Ã, B ~ ) ∑ 

= ⎟ ⎟ 

5 1 ⎛ S( 

V ~ ( u )) − S( 

V ~ ( u )) ⎞ 

= ⎜ A i B i 

1− 

5 i 1 ⎜ 

2 

⎝ 

⎠ 

1 ⎡⎛ 

− 0. 

4 − ( −0. 

2) 

⎞ ⎛ − 0. 

2 − 0 ⎞ ⎛ 0. 

2 − 0. 

4 ⎞ 

= ⎢⎜( 1− 

⎟ + ⎜1− 

⎟ + ⎜1− 

⎟ + 

5 ⎣⎝ 

2 ⎠ ⎝ 2 ⎠ ⎝ 2 ⎠ 

⎛ 0. 

6 − 0. 

6 ⎞ ⎛ 0. 

8 − 0. 

8 ⎞⎤ 

⎜1 

− ⎟ + ⎜1− 

⎟⎥ 

⎝ 2 ⎠ ⎝ 2 ⎠⎦ 

1 

= ( 0. 

9 + 0. 

9 + 0. 

9 + 1+ 

1) 

5 

= 0.94. 

It indicates that the degree of similarity between the vague sets Ã and B ~ is equal to 0.94. 

230

A Review of Biswas’ Methods for Students’ Answerscripts Evaluation 

Biswas (1995) used the matching function S to measure the degree of similarity between two fuzzy sets (Zadeh, 

1965). Let A and B be the vector representation of the fuzzy sets A and B, respectively. Then, the degree of 

similarity S( A , B ) between the fuzzy sets A and B can be calculated as follows (Chen, 1988): 

S( A , B ) = 

A ⋅ B 

, (8) 

Max( 

A ⋅ A, 

B ⋅ B) 

where S( A , B )∈[0, 1]. The larger the value of S( A , B ), the higher the similarity between the fuzzy sets A and B. 

Biswas (1995) presented a “fuzzy evaluation method” (fem) for evaluating students’ answerscripts, based on the 

matching function S. He used five fuzzy linguistic hedges, called Standard Fuzzy Sets (SFS), for students’ 

answerscripts evaluation, i.e., E (excellent), V (very good), G (good), S (satisfactory) and U (unsatisfactory), where 

X = {0%, 20%, 40%, 60%, 80%, 100%}, 

E = {(0%, 0), (20%, 0), (40%, 0.8), (60%, 0.9), (80%, 1), (100%, 1)}, 

V = {(0%, 0), (20%, 0), (40%, 0.8), (60%, 0.9), (80%, 0.9), (100%, 0.8)}, 

G = {(0%, 0), (20%, 0.1), (40%, 0.8), (60%, 0.9), (80%, 0.4), (100%, 0.2)}, 

S = {(0%, 0.4), (20%, 0.4), (40%, 0.9), (60%, 0.6), (80%, 0.2), (100%, 0)}, 

U = {(0%, 1), (20%, 1), (40%, 0.4), (60%, 0.2), (80%, 0), (100%, 0)}. 

He used the vector representation method to represent the fuzzy sets E, V, G, S and U by the vectors E , V ,G , S 

and U , respectively, where 

E = , 

V = , 

G = , 

S = , 

U = . 

Biswas pointed out that “A”, “B”, “C”, “D” and “E” are letter grades, where 0 ≤ E < 30, 30 ≤ D < 50, 50 ≤ C < 70, 

70 ≤ B < 90 and 90 ≤ A ≤ 100. Furthermore, he presented the concept of “mid-grade-points”, where the mid-gradepoints 

of the letter grades A, B, C, D and E are P(A), P(B), P(C), P(D) and P(E), respectively, P(A) = 95, P(B) = 80, 

P(C) = 60, P(D) = 40 and P(E) = 15. Assume that an evaluator evaluates the first question (i.e., Q.1) of the 

answerscript of a student using a fuzzy grade sheet as shown in Table 2. 

Table 2. A fuzzy grade sheet (Biswas, 1995) 

Question No. Fuzzy mark Grade 

Q.1 0.1 0.2 0.3 0.6 0.8 0.9 

Q.2 

Q.3 

… 

… 

… 

… 

… 

… 

… 

Total mark = 

In the second row of Table 2, the fuzzy marks 0.1, 0.2, 0.3, 0.6, 0.8 and 0.9, awarded to the answer of question Q.1, 

indicate that the degrees of the evaluator’s satisfaction for that answer are 0%, 20%, 40%, 60%, 80% and 100%, 

respectively. 

In the following, we briefly review Biswas’ method (1995) for students’ answerscript evaluation as follows: 

Step 1: For each question in the answerscript repeatedly perform the following tasks: 

(1) The evaluator awards a fuzzy mark Fi to each question Q.i and fills up each cell of the ith row for the first seven 

… 

231

columns shown in Table 2, where 1 ≤ i ≤ n. Let F be the vector representation of Fi, where 1 ≤ i ≤ n. 

i 

(2) Based on Eq. (8), calculate the degrees of similarity S( E , F ), S(V , F ), S(G , F ), S( S , F ) and 

S(U , F ), respectively, where E , V , G , S and U are the vector representations of the standard fuzzy sets 

i 

E (excellent), V (very good), G (good), S (satisfactory) and U (unsatisfactory), respectively, and 1 ≤ i ≤ n. 

(3) Find the maximum value among the values of S( E , F ), S(V , F ), S(G , F ), S( S , F ) and S(U , F ). 

Assume that S(V , F ) is the maximum value among the values of S( E , F ), S(V , F ), S(G , F ), S( S , F ) 

i 

i 

i 

i 

i 

and S(U , F ), then award the letter grade “B” to the question Q.i due to the fact that the letter grade “B” 

i 

corresponds to the standard fuzzy set V (very good). If S( E , F ) = S(V , F ) is the maximum value among the 

values of S( E , F ), S(V , F ), S(G , F ), S( S , F ) and S(U , F ), then award the letter grade “A” to the 

i 

i 

i 

i 

i 

question Q.i due to the fact that the letter grade “A” corresponds to the standard fuzzy set E (excellent). 

Step 2: Calculate the total mark of the student as follows: 

n 1 

× ∑ 

= 

Total Mark = [ T ( Q. 

i) 

× P( 

g i )], 

(9) 

100 i 1 

where T(Q.i) denotes the mark allotted to Q.i in the question paper, gi denotes the grade awarded to Q.i by Step 1 of 

the algorithm, P(gi) denotes the mid-grade-point of gi, and 1 ≤ i ≤ n. Put this total score in the appropriate box at the 

bottom of the fuzzy grade sheet. 

Biswas (1995) also presented a generalized fuzzy evaluation method (gfem) for students’ answerscripts evaluation, 

where a generalized fuzzy grade sheet shown in Table 3 is used to evaluate the students’ answerscripts. 

Table 3. A generalized fuzzy grade sheet (Biswas, 1995) 

Derived letter 

Question No. Fuzzy mark grade Mark 

Q.1 

Q.2 

… 

… 

… 

F11 

F12 

F13 

F14 

F21 

F22 

F23 

F24 

… 

… 

… 

i 

i 

g11 

g12 

g13 

g14 

g21 

g22 

g23 

g24 

… 

… 

… 

Total mark = 

In the generalized fuzzy grade sheet shown in Table 3, for all j = 1, 2, 3, 4 and for all i, gij denotes the derived letter 

grade by the fuzzy evaluation method fem for the awarded fuzzy mark Fij and mi denotes the derived mark awarded 

to the question Q.i, where 

4 

1 

mi = × T (Q.i) × ∑ P ( g ij ) , (10) 

400 

j= 

1 

n 

∑ 

i= 

1 

and the Total Mark = m 

. 

i 

i 

i 

i 

i 

m1 

m2 

i 

… 

… 

… 

i 

i 

i 

i 

232

A New Method for Evaluating Students’ Answerscripts Based on the Similarity Measure 

between Vague Sets 

In this section, we present a new method for evaluating students’ answerscripts based on the similarity measure 

between vague sets. Let X be the universe of discourse. We use five fuzzy linguistic hedges, called Standard Vague 

Sets (SVS), for students’ answerscripts evaluation, i.e., E ~ (excellent), V ~ (very good), G ~ (good), S ~ (satisfactory) 

and U ~ (unsatisfactory), where 


E ~ = [0, 0]/0% + [0, 0]/20% + [0, 0]/40% + [0.4, 0.5]/60% + [0.8, 0.9]/80% + 

[1, 1]/100%, 

V ~ = [0, 0]/0% + [0, 0]/20% + [0, 0]/40% + [0.4, 0.5]/60% + [1, 1]/80% + 

[0.7, 0.8]/100%, 

G ~ = [0, 0]/0% + [0, 0]/20% + [0.4, 0.5]/40% + [1, 1]/60% + [0.8, 0.9]/80% + 

[0.4, 0.5]/100%, 

S ~ = [0, 0]/0% + [0.4, 0.5]/20% + [1, 1]/40% + [0.8, 0.9]/60% + [0.4, 0.5]/80% + 


U ~ = [1, 1]/0% + [1, 1]/20% + [0.4, 0.5]/40% + [0.2, 0.3]/60% + [0, 0]/80% + 

[0, 0]/100%. 

Assume that “A”, “B”, “C”, “D” and “E” are letter grades, where 0 ≤ E < 30, 30 ≤ D < 50, 50 ≤ C < 70, 70 ≤ B < 90 

and 90 ≤ A ≤ 100. Assume that an evaluator evaluates the first question (i.e., Q.1) of a student’s answerscript, using a 

vague grade sheet as shown in Table 4. 

Table 4. A vague grade sheet 

Question No. Vague mark 

Q.1 

Q.2 

Q.3 

[0, 0] [0.1, 0.2] [0.3, 0.4] [0.6, 0.7] [0.7, 0.8] [1, 1] 

… 

Q.n 

… 

… 

… 

… 

… 

Total mark = 

… 

Derived letter 

grade 

In the second row of the vague grade sheet shown in Table 4, the vague marks [0, 0], [0.1, 0.2], [0.3, 0.4], [0.6, 0.7], 

[0.7, 0.8] and [1, 1], awarded to the answer of question Q.1, indicate that the degrees of the evaluator’s satisfaction 

for that answer are 0%, 20%, 40%, 60%, 80% and 100%, respectively. Let the vague mark of the answer of question 

~ 

~ 

Q.1 be denoted by F . Then, we can see that F is a vague set of the universe of discourse X, where 

1 

1 


~ 

F = [0, 0]/0% + [0.1, 0.2]/20% + [0.3, 0.4]/40% + [0.6, 0.7]/60% + 

1 

[0.7, 0.8]/80% + [1, 1]/100%. 

The proposed vague evaluation method (VEM) for students’ answerscripts evaluation is presented as follows: 

Step 1: For each question in the answerscript repeatedly perform the following tasks: 

(1) The evaluator awards a vague mark F i 

~ represented by a vague set to each question Q.i by his/her judgment and 

fills up each cell of the ith row for the first seven columns shown in Table 4, where 1 ≤ i ≤ n. 

(2) Based on Eq. (7), calculate the degrees of similarity H( E ~ , F i 

~ ), H(V ~ , F i 

~ ), H(G ~ , F i 

~ ), H( S ~ , F i 

~ ) and 

H(U ~ , F i 

~ ), respectively, where E ~ (excellent), V ~ (very good), G ~ (good), S ~ (satisfactory) and U ~ 

(unsatisfactory) are standard vague sets. 

… 

233

(3) Find the maximum value among the values of H( E ~ , 

H(W ~ , i 

i 

F ~ ) is the largest value among the values of H( E ~ , 

F ~ ), H(V ~ , F ~ ), H(G ~ , F ~ ), H( S ~ , 

i 

i 

F ~ ), H(V ~ , F ~ ), H(G ~ , F ~ ), H( S ~ , 

i 

i 

i 

F ~ ) and H(U ~ , F ~ ). If 

i 

F ~ ) and H(U ~ , 

whereW ~ ∈{ E ~ ,V ~ ,G ~ , S ~ ,U ~ }, then translate the standard vague set W ~ into the corresponding letter grade, 

where the standard vague set E ~ is translated into the letter grade “A”, the standard vague set V ~ is translated 

into the letter grade “B”, the standard vague set G ~ is translated into the letter grade “C”, the standard vague set 

S ~ is translated into the letter grade “D”, and the standard vague set U ~ is translated into the letter grade “E”. 

For example, assume that H(V ~ , F i 

~ ) is the maximum value among the values of H( E ~ , F i 

~ ), H(V ~ , F i 

~ ), 

H(G ~ , F i 

~ ), H( S ~ , F i 

~ ) and H(U ~ , F i 

~ ), then award grade “B” to the question Q.i due to the fact that the letter 

grade “B” corresponds to the standard vague set V ~ (very good). If H( E ~ , F i 

~ ) = H(V ~ , F i 

~ ) is the maximum 

value among the values of H( E ~ , F i 

~ ), H(V ~ , F i 

~ ), H(G ~ , F i 

~ ), H( S ~ , F i 

~ ) and H(U ~ , F i 

~ ), then award the letter 

grade “A” to the question Q.i due to the fact that the letter grade “A” corresponds to the standard vague set E ~ 

(excellent). 

Step 2: Calculate the total mark of the student as follows: 

1 n 

Total Mark = × ~ ~ 

∑ [ T ( Q. 

i) 

× K( 

g ) × H ( w, 

F )], 

(11) 


i 

i 

i = 1 

where T(Q.i) denotes the mark allotted to the question Q.i in the question paper, gi denotes the letter grade awarded 

to Q.i by Step 1, K(gi) denotes the derived grade-point of the letter grade gi based on the index of optimism λ 

determined by the evaluator, where λ∈[0, 1], H(W ~ , F i 

~ ) is the maximum value among the values of H( E ~ , F i 

~ ), 

H(V ~ , F i 

~ ), H(G ~ , F i 

~ ), H( S ~ , F i 

~ ) and H(U ~ , F i 

~ ), W ~ ∈{ E ~ ,V ~ ,G ~ , S ~ ,U ~ }, such that the derived letter grade 

awarded to the question Q.i is gi, and 1 ≤ i ≤ n. If 0 ≤ λ < 0.5, then the evaluator is a pessimistic evaluator. If λ = 0.5, 

then the evaluator is a normal evaluator. If 0.5 < λ ≤ 1.0, then the evaluator is an optimistic evaluator. Assume that 

the derived letter grade obtained in Step 1 with respect to the question Q.i is gi, where gi∈{A, B, C, D, E} and 0 ≤ y1 

≤ gi ≤ y2 ≤ 100, then the derived grade-point K(gi) shown in Eq. (8) is calculated as follows: 

K(gi) = (1 – λ) × y1 + λ × y2, (12) 

where λ is the index of optimism determined by the evaluator, λ∈[0, 1], and 0 ≤ y1 ≤ K(gi) ≤ y2 ≤ 100. Put the 

derived total mark in the appropriate box at the bottom of the vague grade sheet. 

Example 2: Consider a student’s answerscript to an examination of 100 marks. Assume that in total there are four 

questions to be answered: 

TOTAL MARKS = 100, 

Q.1 carries 30 marks, 



Q.4 carries 20 marks. 

Assume that an evaluator awards the student’s answerscript using the vague grade sheet shown in Table 5, where the 

index of optimismλ determined by the evaluator is 0.60, i.e.,λ = 0.60. Assume that “A”, “B”, “C”, “D” and “E” are 

letter grades, where 0 ≤ E < 30, 30 ≤ D < 50, 50 ≤ C < 70, 70 ≤ B < 90 and 90 ≤ A ≤ 100. 

Table 5. Vague grade sheet of Example 2 

Question No. Vague mark Derived letter grade 

Q.1 [0, 0] [0, 0] [0, 0] [0.4, 0.5] [1, 1] [0.5, 0.6] 

Q.2 [0, 0] [0, 0] [0, 0] [0.4, 0.5] [0.8, 0.9] [1, 1] 

Q.3 [0, 0] [0.4, 0.5] [1, 1] [0.6, 0.7] [0.4, 0.5] [0, 0] 

Q.4 [0.8, 0.9] [0.5, 0.6] [0.2, 0.3] [0, 0] [0, 0] [0, 0] 

Total mark = 

i 

i 

F ~ ), 

i 

234

From Table 5, we can see that the vague marks of the questions Q.1, Q.2, Q.3 and Q.4 represented by vague sets 

~ ~ ~ ~ 

F , F , F , respectively, where 

are 1 

~ 

2 

F and 

3 4 

F = [0, 0]/0% + [0, 0]/20% + [0, 0]/40% + [0.4, 0.5]/60% + [1, 1]/80% + 

1 

[0.5, 0.6]/100%, 

~ 

F = [0, 0]/0% + [0, 0]/20% + [0, 0]/40% + [0.4, 0.5]/60% + [0.8, 0.9]/80% + 

2 


~ 

F = [0, 0]/0% + [0.4, 0.5]/20% + [1, 1]/40% + [0.6, 0.7]/60% + [0.4, 0.5]/80% + 

3 


~ 

F 4 = [0.8, 0.9]/0% + [0.5, 0.6]/20% + [0.2, 0.3]/40% + [0, 0]/60% + [0, 0]/80% + 

[0, 0]/100%. 

[Step 1] According to the standard vague sets E ~ ,V ~ ,G ~ , S ~ ,U ~ ~ ~ 

and the vague marks F , 

1 F , 

2 

vague values, as shown in Table 6. 

~ 

F , 

3 4 

~ 

F , we can get the 

Table 6. Vague values of Example 2 

t V~ ( t) 

E V~ ( t) 

V V~ ( t) 

G V~ ( t) 

S V ~ ( t) 

U 

V ~ ( t) 

F1 

V~ ( t) 

F2 

V~ ( t) 

F3 

V~ ( t) 

F4 

0 % [0, 0] [0, 0] [0, 0] [0, 0] [1, 1] [0, 0] [0, 0] [0, 0] [0.8, 0.9] 

20 % [0, 0] [0, 0] [0, 0] [0.4, 0.5] [1, 1] [0, 0] [0, 0] [0.4, 0.5] [0.5, 0.6] 

40 % [0, 0] [0, 0] [0.4, 0.5] [1, 1] [0.4, 0.5] [0, 0] [0, 0] [1, 1] [0.2, 0.3] 

60 % [0.4, 0.5] [0.4, 0.5] [1, 1] [0.8, 0.9] [0.2, 0.3] [0.4, 0.5] [0.4, 0.5] [0.6, 0.7] [0, 0] 

80 % [0.8, 0.9] [1, 1] [0.8, 0.9] [0.4, 0.5] [0, 0] [1, 1] [0.8, 0.9] [0.4, 0.5] [0, 0] 

100 % [1, 1] [0.7, 0.8] [0.4, 0.5] [0, 0] [0, 0] [0.5, 0.6] [1, 1] [0, 0] [0, 0] 

By applying Eq. (5), we can get scores of the vague values, as shown in Table 7. 

Table 7. Scores of the vague values of Example 2 

t S( V~ 

( t)) 

E S( V~ 

( t)) 

V S( V ~ ( t)) 

G S( V~ 

( t)) 

S S( V ~ ( t)) 

U 

S( V ~ ( t)) 

F 1 

S( V ~ ( t)) 

F2 

S( V ~ ( t)) 

F3 

S( V ~ ( t)) 

F4 

0 % -1 -1 -1 -1 1 -1 -1 -1 0.7 

20 % -1 -1 -1 -0.1 1 -1 -1 -0.1 0.1 

40 % -1 -1 -0.1 1 -0.1 -1 -1 1 -0.5 

60 % -0.1 -0.1 1 0.7 -0.5 -0.1 -0.1 0.3 -1 

80 % 0.7 1 0.7 -0.1 -1 1 0.7 -0.1 -1 

100 % 1 0.5 -0.1 -1 -1 0.1 1 -1 -1 

H(X, Y) Y 

X 

Table 8. The degrees of similarity between the vague sets 

~ 

~ 

~ 

F 1 

F F 2 

3 F 4 

E ~ 0.900 1.000 0.492 0.342 

V ~ 0.967 0.942 0.508 0.358 

G ~ 0.792 0.742 0.633 0.350 

S ~ 0.508 0.458 0.967 0.425 

U ~ 0.300 0.250 0.508 0.825 

~ 

235

By applying Eq. (7), we can get the degree of similarity H(X, Y) between the vague values X and Y, where 

~ ~ ~ ~ 

F }, as shown in Table 8. 

X∈{ E ~ ,V ~ ,G ~ , S ~ ,U ~ } and Y ∈{ F 1 , F 2 , F 3 , 3 

Because H(V ~ ~ 

, F 1 ) is the maximum value among the values of H( E ~ ~ 

, F 1 ), H(V ~ ~ 

, F 1 ), H(G ~ ~ 

, F 1 ), H( S ~ ~ 

, F 1 ) 

and H(U ~ ~ 

, F 1 ), we award grade “B” to the question Q.1 due to the fact that the letter grade “B” corresponds to the 

standard vague set V ~ (very good). 

Because H( E ~ ~ 


, F 2 ), H(V ~ ~ 

, F 2 ), H(G ~ ~ 

, F 2 ), H( S ~ ~ 

, F 2 ) 

and H(U ~ ~ 

, F 2 ), we award grade “A” to the question Q.2 due to the fact that the letter grade “A” corresponds to the 

standard vague set E ~ (excellent). 

Because H( S ~ ~ 


, F 3 ), H(V ~ ~ 

, F 3 ), H(G ~ ~ 

, F 3 ), H( S ~ ~ 

, F 3 ) 

and H(U ~ ~ 

, F 3 ), we award grade “D” to the question Q.3 due to the fact that the letter grade “D” corresponds to the 

standard vague set S ~ (satisfactory). 

Because H(U ~ ~ 


, F 4 ), H(V ~ ~ 

, F 4 ), H(G ~ ~ 

, F 4 ), H( S ~ ~ 

, F 4 ) 

and H(U ~ ~ 

, F 4 ), we award grade “E” to the question Q.4 due to the fact that the letter grade “E” corresponds to the 

standard vague set U ~ (unsatisfactory). 

[Step 2] Because 90 ≤ A ≤ 100, 70 ≤ B < 90, 30 ≤ D < 50 and 0 ≤ E < 30, where “A”, “B”, “D” and “E” are letter 

grades, and the index of optimismλ determined by the evaluator is 0.60 (i.e.,λ = 0.60), based on Eq. (12), we can get 

the following results: 

K(A) = (1 – 0.60) × 90 + 0.60 × 100 = 96, 

K(B) = (1 – 0.60) × 70 + 0.60 × 90 = 82, 

K(D) = (1 – 0.60) × 30 + 0.60 × 50 = 42, 

K(E) = (1 – 0.60) × 0 + 0.60 × 30 = 18. 

Because the questions Q.1, Q.2, Q.3 and Q.4 carry 30 marks, 30 marks, 20 marks and 20 marks, respectively, and 

because H(V ~ ~ 

, F 1 ) = 0.967, H( E ~ ~ 

, F 2 ) = 1.000, H( S ~ ~ 

, F 3 ) = 0.967 and H(U ~ ~ 

, F 4 ) = 0.825, based on Eq. (11), the 

total mark of the student is evaluated as follows: 

1 

(30× 82× 0.967 + 30× 96× 1.000 + 20× 42× 0.967 + 20× 18× 0.825) 


1 

= (2378.82 + 2880 + 812.28 + 297) 


= 63.681 

= 64 (assuming that no half mark is given in the total mark). 

A Generalized Method for Evaluating Students’ Answerscripts Based on the Similarity 

Measure between Vague Sets 

In this section, we present a generalized vague evaluation method (GVEM) for students’ answerscripts evaluation 

based on the similarity measure between vague sets, where a generalized vague grade sheet shown in Table 9 is used 

to evaluate the students’ answerscripts. 

Table 9. A generalized vague grade sheet 

Question No. Sub-questions Vague mark Derived letter grade Mark 

Q.1 

Q.11 

~ 

F 11 

~ 

F 

g 11 

g 

12 

Q.12 12 

m1 

236

Q.2 

Q.n 

… 

~ 

Q.13 F g 

13 

13 

~ 

Q.14 F g 

14 

14 

~ 

Q.21 F g 

21 

21 

~ 

Q.22 F g 

22 

22 

~ 

Q.23 F g 

23 

23 

~ 

Q.24 F g 

24 

24 

… 

… 

~ 

Q.n1 F g 

n1 

n1 

~ 

Q.n2 F g 

n2 

n2 

~ 

Q.n3 F g 

n3 

n3 

~ 

Q.n4 F g 

n4 

n4 

… 

Total mark = 

In the generalized vague grade sheet shown in Table 9, each question Q.i consists of four sub-questions, i.e., Q.i1, 

Q.i2, Q.i3 and Q.i4. For all j = 1, 2, 3, 4 and for all i, gij is the derived letter grade by the proposed vague evaluation 

method VEM of the awarded vague mark F ij 

~ with respect to the sub-question Q.ij, and mi is the derived mark 

awarded to the question Q.i, 

1 4 

mi = × T (Q.i) × ~ 

∑ [ K ( gij 

) × H ( w, 

Fij)], 

(13) 

400 

j= 

1 

and 

n 

∑ 

i= 

1 

Total Mark = m . 

i 

where T(Q.i) denotes the mark allotted to Q.i in the question paper, gij denotes the derived letter grade awarded to 

Q.i, and K(gij) denotes the derived grade-point of the letter grade gij based on the index of optimism λ determined by 

the evaluator, whereλ ∈[0, 1], H(W ~ , ij F~ ) is the maximum value among the values of H( E ~ , ij F~ ), H(V ~ , ij F~ ), 

H(G ~ , ij F~ ), H( S ~ , ij F~ ) and H(U ~ , ij F~ ), W ~ ∈{ E ~ ,V ~ ,G ~ , S ~ ,U ~ }, such that the derived letter grade awarded to the 

question Q.ij is gij, 1 ≤ j ≤ 4, and 1 ≤ i ≤ n. If 0 ≤λ < 0.5, then the evaluator is a pessimistic evaluator. Ifλ = 0.5, then 

the evaluator is a normal evaluator. If 0.5



Q.4 carries 30 marks. 

Assume that the index of optimismλ of the evaluator is 0.60 (i.e., λ = 0.60). The evaluator uses Biswas’ method 

(1995) and the proposed method to evaluate the student’s answerscript on different days, respectively. The results are 

shown in Fig. 2 and Fig. 3, respectively. A comparison of the evaluating results of the student’s answerscript is 

shown in Table 10. From Table 10, we can see that the proposed method is more stable to evaluate students’ 

answerscripts than Biswas’ method (1995). It can evaluate students’ answerscripts in a more flexible and more 

intelligent manner. 

July 1, 2006 

Question No. Satisfaction Levels Grade 

Q.1 0 0 0 0.6 0.9 0.8 

Q.2 0 0 0.6 0.9 0.8 0 

Q.3 0 0 0 0.6 0.8 0.9 

Q.4 0 0.6 0.9 0.8 0.2 0 

Total Mark = 

July 2, 2006 


Q.1 0 0 0 0.8 0.9 1 

Q.2 0 0 0.7 0.8 0.9 0 

Q.3 0 0 0 0.7 0.9 0.8 

Q.4 0 0.5 0.8 0.7 0 0 

Total Mark = 

July 3, 2006 


Q.1 0 0 0 0.6 0.9 0.7 

Q.2 0 0 0.6 0.8 0.7 0 

Q.3 0 0 0 0.5 0.7 0.9 

Q.4 0 0.5 0.8 0.6 0 0 

Total Mark = 

July 4, 2006 


Q.1 0 0 0 0.6 0.8 0.7 

Q.2 0 0 0.5 0.9 0.7 0 

Q.3 0 0 0 0.7 0.9 0.8 

Q.4 0 0.6 0.9 0.7 0 0 

Total Mark = 

Figure 2. Evaluating the student’s answerscript at different days using Biswas’ method (1995) 

July 1, 2006 

Question No. Vague marks Grade 

Q.1 [0, 0] [0, 0] [0, 0] [0.6, 0.7] [0.8, 0.9] [0.8, 0.9] 

Q.2 [0, 0] [0, 0] [0.6, 0.7] [0.8, 0.9] [0.8, 0.9] [0, 0] 

Q.3 [0, 0] [0, 0] [0, 0] [0.6, 0.7] [0.8, 0.9] [0.8, 0.9] 

Q.4 [0, 0] [0.5, 0.6] [0.8, 0.9] [0.7, 0.8] [0.1, 0.2] [0, 0] 

Total mark = 

July 2, 2006 


238

Q.1 [0, 0] [0, 0] [0, 0] [0.7, 0.8] [0.8, 0.9] [0.9, 1.0] 

Q.2 [0, 0] [0, 0] [0.6, 0.7] [0.8, 0.9] [0.8, 0.9] [0, 0] 

Q.3 [0, 0] [0, 0] [0, 0] [0.7, 0.8] [0.8, 0.9] [0.8, 0.9] 

Q.4 [0, 0] [0.5, 0.6] [0.8, 0.9] [0.7, 0.8] [0, 0] [0, 0] 

Total mark = 

July 3, 2006 


Q.1 [0, 0] [0, 0] [0, 0] [0.6, 0.7] [0.8, 0.9] [0.7, 0.8] 

Q.2 [0, 0] [0, 0] [0.6, 0.7] [0.8, 0.9] [0.7, 0.8] [0, 0] 

Q.3 [0, 0] [0, 0] [0, 0] [0.5, 0.6] [0.7, 0.8] [0.8, 0.9] 

Q.4 [0, 0] [0.5, 0.6] [0.8, 0.9] [0.6, 0.7] [0, 0] [0, 0] 

Total mark = 

July 4, 2006 


Q.1 [0, 0] [0, 0] [0, 0] [0.6, 0.7] [0.8, 0.9] [0.8, 0.9] 

Q.2 [0, 0] [0, 0] [0.5, 0.6] [0.8, 0.9] [0.7, 0.8] [0, 0] 

Q.3 [0, 0] [0, 0] [0, 0] [0.7, 0.8] [0.8, 0.9] [0.8, 0.9] 

Q.4 [0, 0] [0.6, 0.7] [0.8, 0.9] [0.7, 0.8] [0, 0] [0, 0] 

Total mark = 

Figure 3. Evaluating the student’s answerscript at different days using the proposed method 

Table 10. A comparison of the evaluating results for different methods 

Methods 

Total 

mark 

Days Biswas’ method (1995) The proposed method 

July 1, 2006 69 68 

July 2, 2006 72 68 

July 3, 2006 55 68 

July 4, 2006 55 68 

The Merits of the Proposed Methods 

The proposed methods have the following advantages: 

(1) The proposed methods are more flexible and more intelligent than Biswas’ methods (1995) due to the fact that we 

use vague sets rather than fuzzy sets to represent the vague mark of each question, where the evaluator can use 

vague values to indicate the degree of the evaluator’s satisfaction for each question. Especially, the proposed 

methods are particularly useful when the assessment involves subjective evaluation. 

(2) The proposed methods are more stable to evaluate students’ answerscripts than Biswas’ methods (1995). They 

can evaluate students’ answerscripts in a more flexible and more intelligent manner. 

Conclusions 

In this paper, we have presented two new methods for evaluating students’ answerscripts based on the similarity 

measure between vague sets. The vague marks awarded to the answers in the students’ answerscripts are represented 

by vague sets, where each element belonging to a vague set is represented by a vague value. An index of 

optimismλ determined by the evaluator is used to indicate the degree of optimism of the evaluator, whereλ ∈[0, 1]. 

Because the proposed methods use vague sets to evaluate students’ answerscripts rather than fuzzy sets, they can 

evaluate students’ answerscripts in a more flexible and more intelligent manner. The experimental results show that 

239

the proposed methods can evaluate students’ answerscripts more stable than Biswas’ methods (1995). 


The authors would like to thank Professor Jason Chiyu Chan, Department of Education, National Chengchi 

University, Taipei, Taiwan, Republic of China, for providing very helpful comments and suggestions. This work was 

supported in part by the National Science Council, Republic of China, under Grant NSC 95-2221-E-011-117-MY2. 

References 

Biswas, R. (1995). An application of fuzzy sets in students’ evaluation. Fuzzy Sets and Systems, 74 (2), 187-194. 

Chang, D. F., & Sun, C. M. (1993). Fuzzy assessment of learning performance of junior high school students. Paper 

presented at the First National Symposium on Fuzzy Theory and Applications, June 25-26, 1993, Hsinchu, Taiwan. 

Chen, S. M. (1988). A new approach to handling fuzzy decisionmaking problems. IEEE Transactions on Systems, 

Man, and Cybernetics, 18 (6), 1012-1016. 

Chen, S. M. (1995a). Arithmetic operations between vague sets. Paper presented at the International Joint 

Conference of CFSA/IFIS/SOFT’95 on Fuzzy Theory and Applications, December 7-9, 1995, Taipei, Taiwan. 

Chen, S. M. (1995b). Measures of similarity between vague sets. Fuzzy Sets and Systems, 74 (2), 217-223. 

Chen, S. M. (1997). Similarity measures between vague sets and between elements. IEEE Transactions on Systems, 

Man, and Cybernetics-Part B: Cybernetics, 27 (1), 153-158. 

Chen, S. M. (1999). Evaluating the rate of aggregative risk in software development using fuzzy set theory. 

Cybernetics and Systems, 30 (1), 57-75. 

Chen, S. M., & Lee, C. H. (1999). New methods for students’ evaluating using fuzzy sets. Fuzzy Sets and Systems, 

104 (2), 209-218. 

Chen, S. M., & Wang, J. Y. (1995). Document retrieval using knowledge-based fuzzy information retrieval 

techniques. IEEE Transactions on Systems, Man, and Cybernetics, 25 (5), 793-803. 

Cheng, C. H., & Yang, K. L. (1998). Using fuzzy sets in education grading system. Journal of Chinese Fuzzy 

Systems Association, 4 (2), 81-89. 

Chiang, T .T., & Lin C. M. (1994). Application of fuzzy theory to teaching assessment. Paper presented at the 1994 

Second National Conference on Fuzzy Theory and Applications, September 15-17, 1994, Taipei, Taiwan. 

Frair, L. (1995). Student peer evaluations using the analytic hierarchy process method. Paper presented at the 

Frontiers in Education Conference, November 1-4, 1995, Atlanta, GA, USA. 

Gau, W. L., & Buehrer, D. J. (1993). Vague sets. IEEE Transactions on Systems, Man, and Cybernetics, 23 (2), 610- 

614. 

Echauz, J. R., & Vachtsevanos, G. J. (1995). Fuzzy grading system. IEEE Transactions on Education, 38 (2), 158- 

165. 

Hwang, G. J., Lin, B. M. T., & Lin, T. L. (2006). An effective approach for test-sheet composition with large-scale 

item banks, Computers & Education, 46 (2), 122-139. 

Kaburlasos, V. G., Marinagi, C. C., & Tsoukalas, V. T. (2004). PARES: A software tool for computer-based testing 

240

and evaluation used in the Greek higher education system. Paper presented at the 2004 IEEE International 

Conference on Advanced Learning Technologies, August 30 – September 1, 2004, Joensuu, Finland. 

Law, C. K. (1996). Using fuzzy numbers in education grading system. Fuzzy Sets and Systems, 83 (3), 311-323. 

Liu, C. L. (2005). Using mutual information for adaptive item comparison and student assessment. Educational 

Technology & Society, 8 (4), 100-119. 

Ma, J., & Zhou, D. (2000). Fuzzy set approach to the assessment of student-centered learning. IEEE Transactions on 

Education, 43 (2), 237-241. 

McMartin, F., Mckenna, A., & Youssefi, K. (2000). Scenario assignments as assessment tools for undergraduate 

engineering education. IEEE Transactions on Education, 43 (2), 111-119. 

Nykanen, O. (2006). Inducing fuzzy models for student classification. Educational Technology & Society, 9 (2), 223- 

234. 

Pears, A., Daniels, M., Berglund, A., & Erickson, C. (2001). Student evaluation in an international collaborative 

project course. Paper presented at the First International Workshop on Internet-Supported Education, January 08 - 

12, 2001, San Diego, CA, USA. 

Wang, H. Y., & Chen, S. M. (2006a). New methods for evaluating the answerscripts of students using fuzzy sets. 

Paper presented at the 19 th International Conference on Industrial, Engineering & Other Applications of Applied 

Intelligent Systems, June 27-30, 2006, Annecy, France. 

Wang, H. Y., & Chen, S. M. (2006b). New methods for evaluating students’ answerscripts using fuzzy numbers 

associated with degrees of confidence. Paper presented at the 2006 IEEE International Conference on Fuzzy 

Systems, July 16-21, 2006, Vancouver, BC, Canada. 

Wang, H. Y., & Chen, S. M. (2006c). New methods for evaluating students’ answerscripts using vague values. Paper 

presented at the 9 th Joint Conference on Information Sciences, October 8-11, 2006, Kaohsiung, Taiwan. 

Wang, H. Y., & Chen, S. M. (2006d). Evaluating students’ answerscripts based on the similarity measure between 

vague sets. Paper presented at the 11 th Conference on Artificial Intelligence and Applications, December 15-16, 

2006, Kaohsiung, Taiwan. 

Weon, S., & Kim, J. (2001). Learning achievement evaluation strategy using fuzzy membership function. Paper 

presented at the 31 st ASEE/IEEE Frontier in Education Conference, October 10-13, 2001, Reno, NV, USA. 

Wu, M. H. (2003). A research on applying fuzzy set theory and item response theory to evaluate learning 

performance, Master Thesis, Department of Information Management, Chaoyang University of Technology, Taiwan. 

Zadeh, L. A. (1965). Fuzzy sets. Information and Control, 8, 338-353. 

241

Gogoulou, A., Gouli, E., Grigoriadou, M., Samarakou, M., & Chinou, D. (2007). A Web-based Educational Setting Supporting 

Individualized Learning, Collaborative Learning and Assessment. Educational Technology & Society, 10 (4), 242-256. 

A Web-based Educational Setting Supporting Individualized Learning, 

Collaborative Learning and Assessment 

Agoritsa Gogoulou 1 , Evangelia Gouli 1 , Maria Grigoriadou 1 , Maria Samarakou 2 and 

Dionisia Chinou 1 

1 Department of Informatics & Telecommunications, University of Athens, Greece // {rgog, lilag, gregor}@di.uoa.gr, 

dionisiagr@gmail.com 

2 Department of Energy Technology, Technological Educational Institution of Athens, Greece // marsam@teiath.gr 

ABSTRACT 

In this paper, we present a web-based educational setting, referred to as SCALE (Supporting Collaboration and 

Adaptation in a Learning Environment), which aims to serve leaning and assessment. SCALE enables learners 

to (i) work on individual and collaborative activities proposed by the environment with respect to learners’ 

knowledge level, (ii) participate actively in the assessment process in the context of self-, peer- or collaborativeassessment 

activities, (iii) work with educational environments, embedded or integrated in SCALE, that 

facilitate the elaboration of the activities and stimulate learners’ active involvement, (iv) use tools that support 

the synchronous and asynchronous collaboration/communication and promote learners’ interaction and 

reflection, and (v) have access to feedback components tailored to their own preferences. Also, learners have 

control on the navigation route through the provided activities and feedback components, personalizing in this 

way the learning process. The results revealed from the formative evaluation of the environment are positive and 

encouraging regarding the usefulness of the supported capabilities and tools. 

Keywords 

Learning, Collaboration, Assessment, Feedback, Adaptation 

Introduction 

An emerging trend in education worldwide is a movement of the focus from that of teaching to that of learning and 

from an individualistic and objectivist view of learning to a social constructivism view (Palinscar, 1998; Reigeluth, 

1999; Vosniadou, 2001). The underlying principle behind the social constructivism view of learning is that 

knowledge is constructed by the active interaction of learner with the environment and the idea that the construction 

of knowledge is socially mediated. It is claimed that effective collaboration has proven itself a successful and 

powerful learning method (Soller, 2001). Collaborative learning activities immerse students in challenging tasks or 

questions and enable them to become immediate practitioners and develop higher order reasoning and problem 

solving skills. In this context, collaborative learning is becoming increasingly used and the advent of communication 

technologies has made computer-mediated collaboration possible. 

Along with the learning process, assessment is considered an important component of an educational setting. 

Assessment plays a significant role in helping learners learn when it is interweaved with learning and instruction 

instead of being postponed at the end of the instruction (Shepard, 2000). Moreover, assessment helps students to 

identify what they have already learned, to observe their personal learning progress and to decide how to further 

direct their learning process. As knowledge construction necessitates higher order thinking, new forms of assessment 

are required. Assessment methods such as self-, peer- and collaborative-assessment have been introduced in recent 

years aiming to enhance/promote learning and integrate assessment with instruction. Self-assessment refers to the 

involvement of learners in making judgments about their own work/performance and aims at fostering reflection on 

one’s own learning and work (Sluijsmans et al., 1999). Peer-assessment refers to those activities of learners in which 

they judge and evaluate the work and/or the performance of their peers, while in collaborative-assessment, learners 

and instructor collaborate in order to clarify objectives and standards/criteria, negotiate details of the assessment and 

discuss any misunderstandings that exist (Sluijsmans et al., 1999). Integral part of the assessment process and a key 

aspect of learning and instruction is considered feedback (Mory, 1996). Feedback should guide and tutor learners 

towards the achievement of the underlying learning goals as well as stimulate and cultivate processes like selfexplanation, 

self-regulation and self-evaluation (Chi et al., 1994). 

Cognitive researchers view each individual learner as paramount in mediating learning. The learner becomes the 

focus of the learner-instruction transaction and instructional sequence decisions and options are adapted to individual 






242

learner’s characteristics. Also, various motivational theories emphasize the importance of learner control. Control 

gives individuals the possibility to make choices, to affect outcomes and feel more competent and provokes sustained 

and intense effort (Lepper, 1985). Merrill (1980) asserted that the control of learning needs to be given to learners 

since in that way they have the possibility to learn better how to learn. Educational environments that attempt to 

combine technological learning tools with personalization that caters for individual characteristics and learning 

preferences have the potential to radically alter the landscape of learning. 

In this context, various research efforts and projects focus on the development of web-based learning environments 

that support either (i) individualized learning (Papanikolaou et al., 2003; Stern & Woolf, 2000; Weber & 

Brusilovsky, 2001) by making adjustments in the educational environment in order to accommodate a diversity of 

learner needs and abilities, or (ii) collaborative learning (Rosatelli & Self, 2004; Scardamalia & Bereiter, 1994, 

Vizcaino et al., 2000) by providing various means of dialogue and actions, facilities for students’ selfregulation/guidance, 

etc to support learners in their communication and in the accomplishment of collaborative 

activities, or (iii) assessment (Conejo et al., 2004; Sung et al., 2005) by offering opportunities to learners to identify 

what they have already learned and what they are able to do and to teachers to administer the assessment process. 

In line with the above efforts, we developed a web-based educational setting, referred to as SCALE (Supporting 

Collaboration and Adaptation in a Learning Environment) (available at http://hermes.di.uoa.gr:8080/scale), aiming to 

integrate learning and assessment by offering capabilities for individualized and collaborative learning as well as 

assessment. More specifically, SCALE enables learners to 

• work on individual and collaborative activities which are developed on the basis of contemporary theories of 

learning and proposed by the environment with respect to learners’ knowledge level, 

• participate actively in the assessment process in the context of self-, peer- or collaborative-assessment activities, 

• work with educational environments, embedded or integrated in SCALE, that facilitate the elaboration of the 

activities and stimulate learners’ active involvement, 

• use tools that support and promote the synchronous and asynchronous collaboration/communication, 

• have access to feedback tailored to their own preferences, and 

• have control on the navigation route through the provided activities and feedback components. 

The paper is structured as follows: In the next section, we give an outline of the theoretical foundations that guided 

the development of SCALE. Following, we describe how the learning setting of SCALE is modeled in terms of (i) 

the learning activities, (ii) the feedback components supported and (iii) the learner and group model. The tools as 

well as the environments supporting learning, collaboration and assessment are briefly presented followed by a 

description of the adaptive capabilities of SCALE. The main functionalities of SCALE are outlined through 

exemplary screen shots. Finally, the paper discusses the results of three empirical studies that were conducted in the 

context of the formative evaluation of the environment. The paper ends with the main points of our work and our near 

future plans. 

Theoretical Foundations 

The design principles of SCALE lie on (i) the Activity Theory which is used as a framework for modeling learning 

situations where individualized learning is interweaved with collaborative learning and the concept of activity serves 

as a unit of analysis (Hill et al., 2003), (ii) researchers’ suggestions that assessment should be represented as a tool 

for learning and powerful learning environments should encompass both instruction and assessment (Dochy & 

McDowell, 1997; Shepard, 2000), and (iii) the view that instruction and feedback should be aligned, as much as 

possible, to each individual learner’s characteristics (Jonassen & Grawboski, 1993). 

Central to the Activity Theory is the notion of activity; an activity is seen as a system of human "doing" whereby a 

subject works on an object, by employing mediational tools, in order to attain a desired outcome. Engeström (1987) 

developed an extended model of an activity, which adds the component of community; then adds rules to mediate 

between subject and community and the division of labour to mediate between object and community. That is, rules 

cover both explicit and implicit norms, conventions, and social relations within a community while division of labour 

refers to the explicit and implicit organisation of the community as related to the transformation process of the object 

into the outcome (Kuutti, 1995). In the framework of the SCALE environment, individualized learning is realized by 

243

enabling learner (subject) to work on individual activities with a specific context (object), which results into a 

specific outcome, utilizing various tools (mediational tools), which are considered necessary for the accomplishment 

of the activity (Figure 1a). The collaborative learning is taking place through collaborative activities where learners 

(subject) collaborate, in groups of up to four members (community), in the context of a specific collaborative 

learning activity (object) utilizing various tools (mediational tools) and undertaking specific roles which determine 

the responsibilities and duties of each learner (division of labour) as well as the rules of the collaboration (rules) 

(Figure 1b). 

(a) (b) 

Figure 1. Application of the Activity Model in SCALE 

While traditional assessment focuses on grading and ranking aspects and emphasizes on the need to find out if the 

student knows, understands, or is able to do, the new role of assessment emphasizes on the need to find out what the 

student knows, understands or is able to do. Many researchers suggest that students will learn more if instruction and 

assessment are integrally related and the provision of information about the quality of students’ work as well as 

about what they can do to improve is crucial for maximizing learning (Pellegrino et al., 2001). To this end, 

assessment should be integrated with feedback for permitting learning to become a logical outcome (Taras, 2002) as 

learners need to know what they are trying to accomplish, how close they are coming to the goal and be 

guided/supported towards the achievement of the underlying goal. Moreover, feedback should be aligned, as much as 

possible, to each individual learner’s characteristics, since individuals differ in their general skills, aptitudes and 

preferences for processing information, constructing meaning from it and/or applying it to new situations (Jonassen 

& Grabowski, 1993). Furthermore, self-, peer- and collaborative-assessment are alternatives in assessment that have 

recently received great attention as they are considered as part of the learning process where skills are developed 

(Sluijsmans et al., 1999). Towards this direction, SCALE supports the automatic assessment of the activities, the 

self-, peer- and collaborative-assessment as well as the provision of informative and tutoring feedback components 

tailored to learner’s individual characteristics. 

In the following, we present the model adopted and developed for (i) the representation of the SCALE learning 

setting in terms of the learning activities, (ii) the feedback components supported and (iii) the learner and the group 

model as a whole. 

Modelling Learning Setting in SCALE 

The SCALE learning setting aims to serve learning and assessment by supporting an educational framework, which 

determines the educational function and the educational/didactical approach followed. The educational function 

concerns either learning (knowledge construction) or assessment (ascertainment of learners’ prior knowledge, 

formative assessment or summative assessment) (Figure 2). For the accomplishment of the educational function, the 

learning setting may exploit an educational/didactical approach that better supports and facilitates the educational 

function under consideration (e.g. the educational approach of concept mapping may effectively serve the 

ascertainment of learners’ prior knowledge) and may require the use of a specific educational tool that facilitates the 

realization of the educational/didactical approach (e.g. in case of concept mapping, the COMPASS environment is 

244

used; see section “Tools Supporting Learning and Assessment”). Learner is engaged actively in the learning setting 

by working out activities which have been developed to address and serve the underlying educational functions and 

have been designed on the principles of the underlying educational/didactical approaches. SCALE attempts to 

support and guide learners by providing a framework that includes the feedback components (informative and 

tutoring), the notebooks (described analytically in the following) and the indicators which provide information about 

the elaboration of the activities (e.g. the number of learners that have worked out an activity, the times that learners 

asked for feedback and the type of feedback provided, the number of groups that have worked out a collaborative 

activity). 

Modelling Learning Activities 

Figure 2. The model of the learning setting in SCALE 

An activity in SCALE serves a specific learning goal, which corresponds to fundamental concept(s) of the subject 

matter (Figure 2). The learning goal is further analysed to learning outcomes that may be classified to the 

Comprehension level (Remember + Understand), the Application level (Apply), the Checking-Criticizing level 

(Evaluate), and the Creation level (Analyse + Create) (Gogoulou et al., 2005a). The activity has the so-called action 

framework, which determines the sub-activities that address and realize the outcomes of the activity. The subactivities 

may be individual or collaborative. Each sub-activity addresses learning outcomes that are classified to the 

abovementioned levels. The activities/sub-activities may have different difficulty level and different degree of 

importance for the accomplishment of the underlying goal with respect to the educational function and the addressed 

learning outcomes. In case of a collaborative activity/sub-activity, the action framework also determines the 

collaboration model that learners follow; the collaboration model specifies the number of group members, the role of 

each member and the moderator of the group being responsible for the submission of their common work and the 

coordination of the collaborative process. Depending on the educational function that the activity serves and the 

underlying outcomes, the assessment may be done by one of the following forms: 

• Automatic assessment: In case of activities including closed questions (i.e. multiple choice, true-false, fill-theblank), 

SCALE can automatically assess learner’s answer. Also, the automatic assessment of concept mapping 

activities is supported as these are accomplished by means of the COMPASS environment (see section “Tools 

Supporting Learning and Assessment”). 

245

• Self-, Peer- and Collaborative-assessment: Self-, Peer- and collaborative-assessment are three forms of 

assessment that enable learners to actively participate in the assessment process, to get inspiration from their 

peers’ work, to develop skills such as critical thinking, teamwork, self-monitoring and regulation, etc. These 

forms of assessment are accomplished by means of the PECASSE environment (see section “Tools Supporting 

Learning and Assessment”). 

• Assessment by the teacher: In case none of the above forms is supported, the teacher is responsible to assess the 

activity and inform learner about his/her performance and guide/tutor him/her appropriately. 

Modelling Feedback Components 

Feedback is considered a key aspect of learning and instruction. Characteristics that influence the effectiveness of 

feedback concern the type of feedback, the amount of the provided information as well as the adaptation to learners’ 

individual differences. In this context, multiple informative and tutoring feedback components are provided during 

the elaboration of the activities in SCALE. The informative feedback components (i.e. correctness-incorrectness of 

response and performance feedback) inform learners about their current state; this information is included in the 

learner model, which is maintained by the environment during the interaction. The tutoring feedback components 

aim to tutor/guide learners and are structured in two levels, activity level and sub-activity level. The feedback 

components of the sub-activity level refer to the concepts of the sub-activity under consideration, while at activity 

level, feedback components are more general and address concepts/topics of the activity. The tutoring feedback 

components are associated with various types of knowledge modules (feedback types) and are distinguished in two 

categories: explanatory and exploratory. The explanatory feedback may include knowledge modules such as a 

description or a definition of the concept/topic, and the correct response whilst the exploratory feedback may include 

(i) an image, (ii) an example, (iii) an advice or an instruction on how to proceed, (iv) a question giving students a hint 

on what to think about, (v) a case study, (vi) a similar activity followed by its answer, and (vii) any answers given to 

the specific activity by other learners. 

The different categories and types of knowledge modules aim to serve learners’ individual preferences and to 

cultivate skills such as critical and analytical thinking, ability to compare and combine alternative solutions, etc. In 

any case, the teacher is responsible to design and develop the appropriate knowledge modules of each level, taking 

into account several factors such as the content of the activity/sub-activity under consideration, the difficulty level of 

the specific activity and the addressed learning outcomes. 

Modelling Learner and Group of Learners 

The Learner Model (LM) reflects specific characteristics of the learner and hence it is used as the main source of the 

adaptive behaviour of SCALE. The information held is divided into domain dependent information and domain 

independent information. As far as the domain dependent information is concerned, the LM keeps information about: 

(i) learner’s knowledge level (qualitative and quantitative estimation) with respect to the learning goals and activities 

that s/he has worked on, and (ii) learner’s behaviour during his/her interaction with the environment in terms of the 

number of times that feedback was asked, type of feedback proposed/selected, time spent on an activity, etc. As far 

as the domain independent information is concerned, the LM keeps general information about the learner such as 

username, profession, learner’s preferences on feedback types, last time/date the learner logged on/off. The LM is 

dynamically updated during learner’s interaction in order to keep track of learner’s “current state”. During 

interaction, learners may access their model and see the information held concerning their progress and interaction 

behaviour. Also, they have the possibility to modify their initially declared preferences regarding the types of 

feedback components supported. The externalisation of learner model aims to support the self-regulation and 

reflection processes and enable learner to modify domain independent information kept in LM. The Group Model 

(GM) holds information for the group as a whole. The GM keeps information about the activities that the group has 

elaborated on, the learners constituting the group, the model of collaboration followed during the elaboration of the 

activities and the date/time the group spend on the activity. 

246

Tools Supporting Learning and Assessment 

For the elaboration of an activity as well as for the promotion of learner’s interaction and reflection, SCALE offers 

various tools either embedded or integrated in the environment. In the following, an outline of these tools is given. 

Concept Mapping Environment 

In case the activity/sub-activity concerns a concept mapping task, the COMPASS environment is used (Gouli et al., 

2006b). COMPASS (COncept MaP ASSessment & learning environment) (available at 

http://hermes.di.uoa.gr/compass) is a web-enabled concept mapping learning environment, which aims to assess 

learner’s understanding as well as to support the learning process by employing a variety of concept mapping 

activities, applying a scheme for the qualitative and quantitative estimation of learner’s knowledge and providing 

different informative, tutoring and reflective feedback components, tailored to learner’s individual characteristics and 

needs. 

Depending on the outcomes, the activities may employ different concept mapping tasks, such as the construction of a 

map, the evaluation/correction, the extension and the completion of a given map. The learners may have at their 

disposal a list of concepts and/or a list of relationships to use in the task and/or may be free to add the desired 

concepts/relationships. The provided lists may contain not only the required concepts/relationships but also 

concepts/relationships that play the role of distracters. 

Learner’s concept map may be assessed automatically by COMPASS. The analysis of the map is based on (i) the 

qualitative characterization of the errors aiming to contribute to the qualitative diagnosis of learner’s knowledge (i.e. 

learner’s incomplete understanding/beliefs and false beliefs) and (ii) the quantitative analysis aiming to evaluate 

learner’s knowledge level on the central concept of the map (Gouli et al., 2005). The results derived from the map 

analysis are represented to learners in an appropriate form during the feedback process. The feedback provided in 

COMPASS aims to serve processes of assessment and learning by (i) informing learners about their performance, (ii) 

guiding and tutoring learners in order to identify their false beliefs, focus on specific errors, reconstruct their 

knowledge and achieve specific learning outcomes addressed by the activity/task, and (iii) supporting reflection in 

terms of encouraging learners to “stop and think” and giving them hints on what to think about (Gouli et al., 2006b). 

The adaptive functionality of the feedback process is based on learner’s knowledge level, preferences and interaction 

behaviour and is implemented through (i) the technology of adaptive presentation that supports the provision of 

alternative forms of feedback and feedback components, and (ii) the stepwise presentation of the feedback 

components in the dialogue-based form of feedback. Moreover, COMPASS gives learners the possibility to have 

control over the feedback presentation process by making the desired selections. 

Synchronous and Asynchronous Communication Tools 

In the framework of a collaborative activity, learners communicate in order to exchange their ideas and decide on 

their common answer. They communicate following a collaboration model, either having the same duties or 

undertaking specific roles. All the collaboration/communication is carried out in a written form through synchronous 

or asynchronous means. In case of synchronous communication, learners use the ACT (Adaptive Communication 

Tool) tool (Gogoulou et al., 2005a), which aims to promote the cultivation of cognitive and communication skills 

and guide learners appropriately during their communication. In particular, ACT: 

(i) Adapts the communication with respect to the collaborative learning setting: ACT supports both the free and 

the structured form of dialogue; the structured dialogue is implemented either through sentence openers or 

communication acts. Depending on the learning outcomes addressed by the collaborative activity and the 

model of collaboration followed by the group members, the tool proposes the most suitable form of dialogue 

and type of scaffolding sentence templates (i.e. sentence openers or communication acts) and provides the most 

meaningful and complete set of scaffolding sentence templates adapted with respect to the collaborative 

learning setting. 

(ii) Enables learners to personalize the communication: the tool offers learners the possibility to have control on 

the adaptation by enabling them to negotiate on and select the form of dialogue (i.e. structured versus free 

dialogue) and the type of scaffolding sentence templates they prefer to use and enrich the provided set of 

scaffolding sentence templates with their own ones in order to cover their own “communication” needs. 

247

(iii) Regulates the communication: ACT monitors and analyses the interaction at various levels and provides 

alternative and complementary representations of the interaction analysis results as well as proposes remedial 

actions to guide learners (Gogoulou et al., 2005b). 

In case of asynchronous communication, learners use an asynchronous communication tool, which supports the 

labeling of the messages (e.g. a message may be a proposal, a question, a clarification) and the exchange of work. 

Peer- and Collaborative-Assessment Environment 

PECASSE (Peer- and Collaborative-ASSessment Environment) is a web-based environment that supports self-, peer- 

and collaborative-assessment (Gouli et al., 2006a) (available at http://hermes.di.uoa.gr:8080/pecasse). Learners may 

act as 

• “authors” being able to submit an activity, which has been elaborated either individually or collaboratively, 

• “assessors” being responsible to evaluate (i) their own activity in a brief way or according to specific criteria 

(self-assessment), and/or (ii) the activities submitted by their peers on their own or by collaborating with other 

learners (peer-assessment) or by collaborating with other learners and the instructor (collaborative-assessment), 

• “feedback evaluators” being able to evaluate the quality of the work/feedback, provided by their assessors. 

The assessment process may be carried out in three consecutive rounds at most. Each round involves the following 

steps: (i) activity submission and brief self-assessment, (ii) review of the assigned activities and provision of 

feedback, (iii) collaboration of authors and assessors, evaluation of assessors and revision of the activity submitted in 

the first step. In PECASSE environment, the review process may emphasize on the grading of the activities and/or 

the provision of useful feedback. The provided review/feedback may be structured and recorded either in an 

assessment form or in an assessment letter. 

Notebooks 

The notebooks give learners the possibility to write down their ideas/comments, to characterize them and, if they 

wish, to publish their notes; a note may be characterized as general information, proposal/answer, question, 

clarification, reasoning, comment or guideline. In this way, the notebooks aim to serve learners’ indirect 

collaboration by enabling them to read and answer the published notes and also to foster processes of reflection, and 

cultivate metacognitive skills such as self-regulation and self-control. 

SCALE supports two types of notebooks at two different levels. At the level of the subject matter, learners have 

access to the Notebook of the Subject Matter on which they maintain personal notes and access/reply/comment notes 

published by others concerning the specific subject matter and the concepts within the subject matter. At the activity 

level, learners have at their disposal the Notebook of the Activity, on which they can maintain personal notes and 

access published notes for the specific activity. This notebook acts as an asynchronous mean for learners’ 

communication in the context of individual activities, aiming to encourage the externalization of personal thoughts 

and argumentation on learners’ beliefs. 

Adaptation in SCALE 

In SCALE, a navigation route through the provided activities and feedback is proposed, based on learner’s 

knowledge level and preferences respectively. Learners’ navigation is supported by using a graphical icon to point 

out the recommended activities and feedback components. Such a personalization aims to support learner in 

achieving the underlying learning goals following his/her own progress. The learner has the possibility to ignore the 

system’s recommendations and follow his/her navigation route. 

The technology of adaptive link annotation is used in order to generate a sequence of activities and feedback 

components that gradually guide learners to accomplish specific activity-related learning outcomes, and finally meet 

the underlying learning goal. In particular, SCALE plans the delivery of the activities for a particular learner (in the 

context of a learning goal), based on his/her progress with respect to the educational function served by the activity 

248

and its difficulty level. For example, if there is an activity aiming to ascertain/assess students’ prior knowledge, then 

it is the first one recommended as proposed by the environment (see Figure 3). Once learner completes such an 

activity, and his/her knowledge level is determined both quantitatively and qualitatively, the adaptation mechanism 

determines the next in sequence proposed activity with respect to learner’s knowledge level and the difficulty level 

of the provided activities. This rule is by-passed if there is an activity that has been defined as proposed by the 

teacher. The last proposed activity within a learning goal is the one (if any) that aims to draw conclusions about the 

degree of achieving the expected learning outcomes (i.e. summative assessment). 

For the delivery of the supported tutoring feedback components, SCALE takes into account learner’s preferences and 

the delivery sequence defined by the teacher. More specifically, initially the adaptation mechanism checks for 

feedback components compatible to learner’s preferences (i.e. whether the types of feedback that learner prefers 

coincide with the types of the available feedback). For a specific feedback type, the sequence of the proposed 

feedback components is determined with respect to the delivery sequence proposed by the teacher (e.g. in case three 

examples are available, these are proposed according to the teacher’s defined order). If learner’s preferences have 

been fulfilled, the rest feedback components are recommended with respect to the delivery sequence concerning the 

rest available feedback types (e.g. first the definition, then the examples and third the correct answer). 

As it is considered essential to allow learners to play an active role and take control over their own learning in order 

to meet their needs and preferences, SCALE gives learners the possibility to have control over the activities and 

feedback components presented by selecting the preferred activity to work out as well as the desired feedback 

component. 

Working with SCALE 

Based on the learning goal that the learner selects (i.e. the learning goal corresponds to fundamental concepts of the 

underlying subject matter), SCALE provides various activities. Figure 3 is the main screen of the SCALE 

environment showing information for the Subject Matter that the learner has chosen. More specifically, the Subject 

Matter “Informatics for Secondary Education” consists of two learning goals; learning goal A (Computer 

Architecture) and learning goal B (Internet). Learning goal A includes two activities A1 and A2. Both activities are 

based on the concept mapping approach (i.e. didactical approach), are individual (i.e. type of activity), are assessed 

automatically by the system and have not yet been submitted by the learner under consideration (i.e. status). The 

learning goal B includes five activities. Activities B1 and B2 are individual, are assessed automatically by the 

system, include one sub-activity consisting of various questions and have not yet been worked out by the learner; 

activity B3 is individual, includes two sub-activities consisting also of questions, is assessed by the teacher and the 

system (i.e. part(s) of the questions are automatically assessed by the system while other parts need to be assessed by 

the teacher) and has not yet been worked out; activities B4 and B5 are collaborative, include only one sub-activity 

assessed by the teacher and have not yet been worked out. According to the adaptation framework, activity B1 is the 

one proposed to the learner (i.e. it is denoted by an icon) as it is an activity aiming to ascertain learner’s prior 

knowledge on the specific learning goal. 

Once learner selects an activity to work on, the corresponding sub-activities are presented. Figure 4 presents one of 

the sub-activities of the activity B3 (Figure 3). The difficulty level of the specific sub-activity is 2 (out of 5), it is 

individual and it consists of a question, asking learner to answer to a multiple-choice question and reason his/her 

answer. The answer given to the multiple-choice question is automatically assessed while the answer given as 

reasoning has to be assessed by the teacher. While working on the activity, learner may have access to the Learner 

Model, the Educational Tools required for the elaboration of the activity, the Notebook of the activity in order to 

record any personal notes or to “communicate” with other learners and the Activity Indicators. Support to learner is 

provided through the Learner Assistant, which presents the feedback available at activity level. Once learner submits 

his/her answer to the sub-activity, the feedback available at sub-activity level is accessible (i.e. an icon similar to the 

Learner Assistant icon appears to the corresponding Feedback column of Figure 4). Figure 5 presents the available 

feedback for the sub-activity depicted in Figure 4. Three types of feedback components are provided: an 

instruction/hint, a case study and a similar problem. The feedback components are proposed according to learner’s 

preferences and the sequence defined by the teacher. Learner ignores the recommendation of the system for feedback 

components (i.e. the case study) and has selected to see the first feedback component providing an instruction/hint. 

249

Figure 3. A screen shot of the SCALE environment showing two learning goals for the Subject Matter “Informatics 

for Secondary Education” 

Figure 4. A screen shot of SCALE showing a sub-activity of the activity B3 presented in Fig 3 

250

Figure 5. A screen shot of the feedback window showing the available feedback components for the sub-activity 

of Figure 4 

Formative Evaluation 

In the context of the formative evaluation of the environment, three empirical studies were conducted at the Department 

of Informatics and Telecommunications of the University of Athens: 

• The 1 st empirical study was conducted during the spring-semester of the academic year 2004-2005 in the context 

of the postgraduate course of “Distance Education and Learning”. The study focused on usability issues of the 

interface, the provided facilities and tools. 

• The 2 nd empirical study was conducted during the winter-semester of the academic year 2005-2006 in the 

context of the undergraduate course of “Didactics of Informatics”. The study focused on usability issues 

regarding the PECASSE environment and students’ attitude towards the peer-assessment process. 

• The 3 rd empirical study was conducted during the spring-semester of the academic year 2005-2006 in the context 

of the undergraduate course of “Informatics in Education” and the postgraduate course of “Distance Education 

and Learning”. The study focused on the structure and presentation of the activities, the provision of feedback 

and the adaptive capabilities of the environment. 

All three studies were qualitative aiming to elicit students’ point of view for various functionalities supported by SCALE 

and the tools embedded or integrated in the environment. 

Process 

1 st study: The 1 st study was carried out through questionnaires including closed and open questions asking students to 

comment and reason their point of view. Thirty-eight students participated in the study, coming from a range of 

backgrounds and having different expertise in the use of web-based learning environments. The study took place in the 

main laboratory of the department and lasted 4 hours. Each student worked on his/her own computer on different 

scenarios. In particular, the working sheet had an activity-oriented behaviour aiming to involve students in different 

functions supported by SCALE; the purpose of the first and the second scenario was to enable students to explore the 

presentation/structure of the activities and the way of working out an activity; the third scenario focused on the elaboration 

of a collaborative activity using the ACT tool; the fourth scenario attempted to investigate the usefulness of the facilities 

provided by COMPASS environment and thus engaged students in a concept mapping task. 

251

2 nd study: Thirty-five students participated in the study, which lasted nine weeks in total. The students had to work on a 

self- and peer-assessment activity provided in SCALE. Learners were asked to design a lesson plan for a specific topic 

(half of them worked on the topic “Internet and search engines” and the rest worked on the topic “The concept of variable 

in programming”). The accomplishment of the activity was supported by the PECASSE environment. Initially students 

submitted their work and they self-evaluated and gave mark to their own work. Following, they were assigned two 

activities to assess: one addressing the same topic as their own and the second one addressing the alternative topic. The 

review process was carried out through an assessment form. As last step, the students received two anonymous reviews 

for their activity and evaluated their assessors. Upon the completion of the whole activity, students were asked to fill and 

submit a questionnaire concerning the evaluation of the PECASSE environment. Also, the students were asked to 

comment on the interface of SCALE and on the capability of the environment to support both the learning and assessment 

processes and provide various tools in order to serve this purpose. 

3 rd study: In the framework of the undergraduate course “Informatics & Education” and the postgraduate course “Distance 

Education and Learning”, eighteen students were asked to act as designers for the development of educational material for 

SCALE. In particular, the students had to explore the environment (i.e. an indicative set of activities had been developed) 

and to design and develop material, following the principles of the environment, for (i) the topic “Looping constructs in 

programming” for the secondary education (the undergraduate students) and (ii) the main concepts (e.g. open education, 

distance education, the role of teacher) of the “Distance Education and Learning” course (the postgraduate students). The 

students had to submit a set of individual and collaborative activities accompanied with appropriate feedback components. 

They also had to comment on the SCALE environment regarding the structure and presentation of the activities, the 

capability of providing alternative feedback types and the adaptation of the environment. 

Results 

The three empirical studies revealed positive and interesting results, which are presented in the following in terms of 

the issues investigated. 

Support of learning and assessment 

The students, who participated in the three studies, found interesting the capability of the environment to support 

both learning and assessment. The variety of activities/sub-activities that students may work on as well as their active 

participation in the learning and assessment process was high in most of the students favour. They rated positively 

the capability of automatic assessment and the provision of immediate informative feedback (i.e. knowledge of 

correctness/incorrectness of their response). It is worthwhile mentioning that their opinion was that the SCALE 

environment can effectively support the instruction process in higher education. 

Despite the positive comments, the students of the 3 rd study acting as designers of educational material for the 

environment, had difficulties in organizing/structuring their activities to the design principles of SCALE. In 

particular, they found hard the decomposition of an activity into a logical structure of sub-activities and the 

specification of characteristics such as difficulty level and outcome level. It seems that their difficulty is mainly 

attributed to their inexperience in acting as authors of educational material. 

Provision of feedback 

As mentioned above, the students considered essential the provision of informative feedback. Furthermore, they 

found very useful the provision of different types of feedback components (e.g. examples, case studies, hints). 

Especially the students of the 3 rd activity, fully explored the alternative feedback components that were available and 

tried to design feedback material to cover all types. They also found useful and quite guiding the structuring and 

provision of feedback at activity and sub-activity level. However, some students claimed that they should have 

access to the feedback of the sub-activity level while working with the specific sub-activity; in the current version of 

the system, the feedback at the sub-activity level is accessible once they submit their answer for first time. 

Support of adaptation 

The students of the 3 rd study took advantage of the capability of the environment to adapt the delivery sequence of 

the available feedback components to their preferences, kept in the learner model. They asserted that the use of an 

indicative icon to point out the recommended feedback component facilitates learner’s navigation while 

simultaneously enables learner to have control on the navigation route and select the desired component. The 

252

adaptivity supported for the recommendation of the most appropriate activity with respect to the students’ progress 

needs to be investigated in a future study, as it was not included in the presented studies. 

Presentation/Structure/Accessibility of the activities & sub-activities 

Most of the students that participated in the three studies expressed their satisfaction regarding the organization/structure 

of the activities. They also marked as adequate the characteristics presented for the activities and sub-activities (Figure 3 

and 4) and they believe that the presented characteristics depict a reasonable amount of information and facilitate their 

interaction. Students’ suggestions concerned the difficulty level of the sub-activities; most of them asked for more details 

for the specific characteristic. Regarding the way of accessing and working on the sub-activities, the students participated 

in the 1 st study reported that they should have access to any sub-activity included in an activity instead of following the 

sequential order imposed by the environment; the corresponding version of the system restricted students to follow a 

sequential order while the current version of the system enables students to select and work on the sub-activities 

following their preferred order. The students of the 2 nd and the 3 rd study commented positively on this change mentioning 

that it is quite helpful to have a look at the content of the sub-activities and subsequently decide on which one to work. 

Moreover, the possibility to submit their answer as many times as they wish stood high in most of the students favour; the 

teacher is responsible to specify for every activity the maximum number of times that students are allowed to access the 

activity and re-submit their answer. 

Facilities and tools supported 

Notebooks: Most students (85%) of the 1 st study that used the notebooks in a systematic way, believed that this facility 

can help in the elaboration of the activities as they have the chance to “collaborate”, exchange their ideas, ask questions, 

externalize their thoughts and share their expertise. Most of them appreciated the participation of the teacher in the 

notebook of the Subject Matter; during the 1 st study, the teacher asked students to express their point of view for the 

variety of activities provided using the corresponding notebook of the Subject Matter and she kept track of the students’ 

notes and participated in the “conversation”. Despite the students’ positive attitude towards the specific facility, a lot of 

them found the way of working with notebooks as moderate (62%) as they considered limited the space provided for 

writing a note and for presenting the list of the submitted notes (these comments were taken into account in the 

development of the current version of the environment). 

ACT tool: In the context of the 1 st study, the students worked on a collaborative activity. For the elaboration of the 

specific activity, the students had to use the ACT tool in order to collaborate and communicate with their partner 

synchronously. All of the students asserted that they had no difficulties in accessing ACT. As far as the evaluation of the 

ACT tool is concerned, the analysis of the students’ answers revealed that a considerable number of students (83%) 

characterized the way of working with the provided scaffolding sentence templates as easy. The majority of the students 

(83%) considered the capability of the ACT tool to group messages into sub-trees and to represent the dialogue in a visual 

graphical form (Dialogue Tree) very useful because it enables them to monitor the dialogue in an organized and 

comprehensive manner, to evaluate the collaboration process more easily and to proceed to interventions in order to 

improve their participation. Only a small number of students (17%) mentioned that there was no need to consult the 

Dialogue Tree during the elaboration of the activities. As far as the adaptation of the scaffolding sentence templates is 

concerned, the majority of the students (80%) considered the provided type of scaffolding sentence templates (i.e. 

sentence openers) appropriate for the corresponding context of the activity and most of them (66%) characterized the 

facility of enriching the predefined sets of sentence openers with their own ones as useful (50% of them took advantage 

of the specific facility during the elaboration of the activities). 

COMPASS environment: COMPASS was used for the elaboration of a concept mapping task in the context of the 1 st 

study. The students accessed the environment through SCALE and used it for the construction of a concept map. While 

working, they used facilities for the analysis of the map, the provision of feedback and the quantitative and qualitative 

estimation of learners’ knowledge level. Most of the facilities were characterized as useful: 68% for the analysis of the 

map, 81% for the provision of feedback, 31% for the quantitative estimation of learner’s knowledge level and 56% for 

the diagnosis of students’ false beliefs and incomplete understanding. A considerable number of students (69%) 

characterized the facility concerning the quantitative estimation of learner’s knowledge level as neutral as they believed 

that the added value in such a tool is the provision of feedback, which helps learners to identify their weaknesses and 

errors and improve their concept maps. 

PECASSE environment: The students of the 2 nd study that involved in the self- and peer-assessment process through the 

PECASSE environment, believe that the peer-assessment process promotes and enhances learning but the majority of 

253

them characterized it as time and effort consuming. A considerable number of students considered that PECASSE fulfils 

the aims of the peer-assessment process, facilitates the execution of the steps and contributes positively in the realization 

of the process in a useful/easy way. They found most of the provided facilities useful and usable and they suggested 

improvements for the management and the completion of the assessment form. Also, 76% of the students believe that 

PECASSE can be incorporated effectively as an assessment tool in the instruction process and about 60% of the students 

were willing to work out activities through PECASSE in the future. As far as the review process is concerned, a 

considerable number of students (89%) were satisfied and considered that the feedback they received was useful and 

helped them to revise their initial activity. 

Conclusions and Outlook 

The educational setting presented in this paper attempts to interweave individualized learning with collaborative 

learning as well as assessment. SCALE supports learning and assessment by (i) enabling learners to select the 

desired learning goal and the activities serving this goal, (ii) providing multiple informative and tutoring feedback 

components both at the activity and the sub-activity level, (iii) supporting various tools, which facilitate the 

elaboration of the activities and support learner’s synchronous and asynchronous communication/collaboration and 

the processes of reflection and self-regulation, and (iii) serving various forms of assessment such as the automatic 

assessment of the activities, the self-, peer- and collaborative-assessment. Moreover, SCALE supports the individual 

learner in achieving the underlying learning goals by proposing a navigation route through the provided activities 

and feedback, based on learner’s knowledge level and preferences respectively. So far, the results of the three studies 

that were carried out revealed that the provided facilities and tools may facilitate and support learning and 

assessment and stood high in most of the students favour. However, the use of the SCALE learning setting in real 

classroom conditions under long periods of time is considered necessary. 


The above research work was partly co-funded by the European Social Fund and National Resources - (EPEAEK II) 

ARXIMHDHS. 

References 

Chi, M., de Leeuw, N, Chiu, M-H., & Lavancher, C. (1994). Eliciting self-explanation improves understanding. 

Cognitive Science, 18, 439–477. 

Conejo, R. Guzmán, E. Millán, E., Trella, M., Pérez-De-La-Cruz, J., & Ríos, A. (2004). SIETTE: A Web-Based Tool 

for Adaptive Testing. International Journal of Artificial Intelligence in Education, 14, 29-61. 

Dochy, F., & McDowell, L. (1997). Assessment as a tool for learning. Studies in Educational Evaluation, 23 (4), 

279-298. 

Engeström, Y. (1987). Learning by expanding: An activity theoretical approach to developmental research, 

Helsinki, Finland: Orienta Konsultit Oy. 

Gogoulou, A. Gouli, E., Grigoriadou, M., & Samarakou, M. (2005a). ACT: A Web-based Adaptive Communication 

Tool. In Koschmann, T., Suthers, D. & Chan, T.W. (Eds.), Proceedings of Computer Supported Collaborative 

Learning 2005: The Next 10 Years!, Mahwah, NJ: Lawrence Erlbaum, 180-189. 

Gogoulou, A. Gouli, E., & Grigoriadou, M. (2005b). Analysing Learner Interaction in an Adaptive Communication 

Tool. Paper presented at the AIED2005 Workshop on Representing and Analyzing Collaborative Interactions: What 

works? When does it work? To what extent?, July 18-22, 2005, Amsterdam. 

254

Gouli, E., Gogoulou, A., Papanikolaou, K., & Grigoriadou, M. (2005). Evaluating Learner’s Knowledge level on 

Concept Mapping Tasks. In Goodyear, P., Sampson, D.G., Yang, D., Kinshuk, Okamoto, T., Hartley, R. & Chen, N- 

S (Eds.), Proceedings of the 5th IEEE International Conference on Advanced Learning Technologies, Los Alamitos: 

IEEE Computer Society, 424-428. 

Gouli, E., Gogoulou, A., & Grigoriadou, M. (2006a). Supporting Self- Peer- and Collaborative-Assessment through a 

Web-based Environment. In Kommers, P. & Richards, G. (Eds.), Proceedings of ED-MEDIA 2006, Chesapeake, 

VA: AACE, 2192-2199. 

Gouli, E., Gogoulou, A., Tsakostas, C., & Grigoriadou, M. (2006b). How COMPASS supports multi-feedback forms 

& components adapted to learner’s characteristics. In Cañas, A. & Novak, J. (Eds.), Proceedings of the Second 

International Conference on Concept Mapping, Universidad de Costa Rica, Vol.1, 255-262. 

Hill, C.M., Cummings, M., & van Aalst, J. (2003). Activity Theory as a Framework for Analyzing Participation 

within Knowledge Building Community. Paper presented at the annual meeting of the American Educational 

Research Association, Chicago, IL, retrieved October 15, 2007, from http://www.educ.sfu.ca/kb/Papers/ 

Hill_Cummings.pdf. 

Jonassen, D., & Grabowski, B. (1993). Handbook of Individual Differences, Learning and Instruction, Hillsdale, NJ: 

Lawrence Erlbaum. 

Kuutti, K. (1995) Activity Theory as a Potential Framework for Human-Computer Interaction Research. In Nardi, 

B.A. (Ed.), Context and Consciousness: Activity Theory and Human-Computer Interaction, MIT Press, 17-44. 

Lepper, M.R. (1985). Microcomputers in education: Motivational and social issues. American Psychologist, 40, 1-18. 

Merrill, M.D. (1980). Learner control in Computer Based Learning. Computers & Education, 4, 77-95. 

Mory, E. (1996). Feedback Research. In Jonassen, D. H. (Ed.), Handbook of research for educational 

communications and technology, New York: Simon & Schuster Maxmillan, 919-956. 

Palinscar, A. (1998). Social constructivist perspectives on teaching and learning. Annual Review of Psychology, 49, 

345-375. 

Papanikolaou, K., Grigoriadou, M., Kornilakis, H., & Magoulas, G. (2003). Personalizing the interaction in a webbased 

educational hypermedia system: the case of INSPIRE. User-Modeling and User-Adapted Interaction, 13 (3), 

213-267. 

Pellegrino, J., Chudowsky, N., & Glaser, R. (2001). Knowing what students know: The Science and Design of 

Educational Assessment, Washington DC: National Academy Press. 

Reigeluth, C.M. (1999). Instructional design theories and models: A new paradigm of instructional theory, vol. II, 

Mahwah, NJ: Lawrence Erlbaum. 

Rosatelli, M., & Self, J. (2004). A Collaborative Case Study System for Distance Learning. International Journal of 

Artificial Intelligence in Education, 14, 97-125. 

Scardamalia, M., & Bereiter, C. (1994). Computer support for knowledge-building communities. The Journal of the 

Learning Sciences, 3 (3), 265-283. 

Shepard, L. (2000). The Role of Assessment in a Learning Culture. Educational Researcher, 29 (7), 4-14. 

Sluijsmans, D., Dochy, F., & Moerkerke, G. (1999). Creating a learning environment by using self- peer- and coassessment. 

Learning Environments Research, 1, 293-319. 

255

Soller, A. (2001). Supporting Social Interaction in an Intelligent Collaborative Learning System. International 

Journal of Artificial Intelligence in Education, 12, 40-62. 

Stern, M.K., & Woolf, B.P. (2000). Adaptive content in an online lecture system. Lecture Notes in Computer 

Science, 1892, 227–238. 

Sung, Y-T., Chang, K-E., Chiou, S-K., & Hou, H-T. (2005). The design and application of a web-based self- and 

peer-assessment system. Computers & Education, 45, 187-202. 

Taras, M. (2002). Using assessment for learning and learning for assessment. Assessment & Evaluation in Higher 

Education, 27 (6), 501-510. 

Vizcaíno, A. Contreras, J., Favela, J., & Prieto, M. (2000). An Adaptive, Collaborative Environment to Develop 

Good Habits in Programming. Lecture Notes in Computer Science 1839, 262-271. 

Weber, G., & Brusilovsky, P. (2001). ELM-ART: An Adaptive Versatile System for Web-based Instruction. 

International Journal of Artificial Intelligence in Education, 12 (4), 351-384. 

Vosniadou, S. (2001). How children learn, retrieved October 15, 2007 from http://www.ibe.unesco.org/publications/ 

EducationalPracticesSeriesPdf/prac07e.pdf. 

256

Roberts, T. S., & McInnerney, J. M. (2007). Seven Problems of Online Group Learning (and Their Solutions). Educational 


Seven Problems of Online Group Learning (and Their Solutions) 

Tim S. Roberts and Joanne M. McInnerney 

Faculty of Business and Informatics, Central Queensland University, Australia // t.roberts@cqu.edu.au // 

cowlrick@optusnet.com.au 

ABSTRACT 

The benefits of online collaborative learning, sometimes referred to as CSCL (computer-supported collaborative 

learning) are compelling, but many instructors are loath to experiment with non-conventional methods of 

teaching and learning because of the perceived problems. This paper reviews the existing literature to present 

the seven most commonly reported such problems of online group learning, as identified by both researchers 

and practitioners, and offers practical solutions to each, in the hope that educators may be encouraged to “take 

the risk”. 

Keywords 

Online collaborative learning, CSCL, Group learning, Group work, Free riders 

Introduction 

The importance and relevance of social interaction to an effective learning process has been stressed by many 

theorists, from Vygotsky (1978), through advocates of situated learning such as Lave and Wenger (1991), and many 

other recent researchers and practitioners. Indeed, the academic, social, and psychological benefits of group learning 

in a face-to-face environment are well documented (see, for example, Johnson & Johnson, 1977, 1984; Slavin, 1987; 

Tinzmann et al, 1990; Bonwell & Eison, 1991; Felder & Brent, 1994; Panitz & Panitz, 1998; Burdett, 2003; Graham 

and Misanchuk, 2004; Roberts, 2004, 2005). 

Online group learning, sometimes referred to as computer-supported collaborative learning (CSCL), if implemented 

appropriately, can provide an ideal environment in which interaction among students plays a central role in the 

learning process (see, for example, Koschmann, 1999, 2001; Lipponen, 2002; Lipponen et al, 2002). Why then is 

online group learning not more widely practiced, particularly within higher education? There are a variety of 

possible reasons that could be supported with some justification. Certainly one reason that would be prominent in 

any list would be educators’ fears of veering away from the well established “sage on the stage” mentality 

(characterised by the traditional lecture / seminar / tutorial format, with notes and other resources provided on the 

Web) to the more increasingly common “guide on the side” mentality (characterised by various forms of group and 

peer learning). These fears can, however, be readily allayed by a prior knowledge of the problems likely to be 

encountered, and appropriate solutions that can be applied. 

The Seven Problems 

Amongst the problems that are thought to be inherent to this method of teaching, the seven most commonly found in 

the literature are the following: 

Problem #1: student antipathy towards group work 

Problem #2: the selection of the groups 

Problem #3: a lack of essential group-work skills 

Problem #4: the free-rider 

Problem #5: possible inequalities of student abilities 

Problem #6: the withdrawal of group members, and 

Problem #7: the assessment of individuals within the groups. 

Many of these problems of online group learning are inter-related. For example, student antipathy (#1) may lead to 

free-riders within groups (#4), and even the withdrawal of some group members (#6), and this in turn may cause 

problems for the assessment of individuals within the groups (#7). Indeed, it will be seen that problem #7 in 

particular is central. Nevertheless, it is perhaps advantageous to examine each of these problems independently, 






257

making reference to the others where appropriate. All of the listed problems are relatively easy to overcome, and the 

possible solutions undemanding when measured in terms of time and resources. 

Problem #1: student antipathy towards group work 

Some students do not care for the idea of group work and can be apathetic, or even on occasions actively hostile to 

the whole idea. Why should this be so? Given that it is relatively commonplace for students to voluntarily congregate 

in groups to discuss assignment problems and solutions outside of class times, it would seem especially surprising. 

Commonly expressed student views against involvement in group work include: 

• I study best on my own 

• I have no need to work in a group 

• I can’t spare the time to meet and communicate with others 

• Others in the group are less capable 

Although some students may be genuinely concerned, experience shows that if their initial antipathy can be 

overcome, many will come to appreciate the advantages that a group learning environment can provide. So how can 

the antipathy best be overcome? 

Solution #1.1: Tell the students the benefits! 

Among the potential benefits which educators should stress to students are the social, psychological, and learning 

benefits, the much greater chance of being received appreciatively by potential employers, and the fact that much of 

their future careers will almost certainly involve working in groups with a diverse range of people who will have a 

wide variety of skills and abilities. Experience of working with others of differing backgrounds and capabilities is 

therefore likely to be highly beneficial. 

The Teaching & Assessment Network (1999: p5) members highlighted the point that 

‘…. departments needed to communicate the purpose of group activity so that students understand the 

associated benefits (and limitations). Such explicitness was seen as an important stage in helping 

students better understand and develop their group work skills.’ 

A minimal list of generic skills honed by the use of group work would include the abilities to cooperate with others; 

to communicate effectively; to lead, and work effectively in, teams; and to organise, delegate, and negotiate. 

Students should also be made aware that any number of surveys point to employers regarding generic skills (perhaps 

most particularly the ability to work effectively in teams) as being of prime importance when selecting graduates. 

Indeed, in many areas, such skills are seen as being equally as important as content knowledge. 

Levin (2002a: p5) indicated that, when it came time for them to partake in real world employment, students involved 

in group learning would have developed the skills of… 

• developing rapport with others 

• negotiating a framework for working with others 

• generating and sustaining motivation and commitment to working together 

• standing back from the hurly-burly of teamwork and 

• making sense of what is going on in one’s team 

• coping with stressful situations that arise 

• evaluating the working of one’s team 

• recognising and making the most of individuals’ dispositions to prefer particular team roles 

• building up one’s teamwork expertise. 

But the development of generic skills is not the only benefit of group learning. Research shows that many students, 

particularly including weaker students, and some minority groups, often learn more effectively in such an 

258

environment. And, in an online setting, the use of group work can greatly reduce the feeling of isolation experienced 

by many students, even the most successful ones. 

Most students are naturally ignorant of the benefits of group learning. They may find out for themselves during the 

learning process, but initially at least, they need to be told! 

Solution #1.2: Make the assessment criteria explicit. 

Students are naturally wary of any system which might judge them based on the merits (or shortcomings) of others. 

It is not sufficient to have a fair assessment system in place – it must, in addition, be made absolutely explicit, right 

from the outset of the course, so that students are fully aware of the basis upon which their individual mark will be 

based. In an online course, it is usually convenient to have this information available on a prominent web page, 

though it could also be communicated via discussion lists, or by individual email. 

The exact nature of how students can be assessed in an online group setting is dealt with in detail in problem #7. The 

important point is that when appropriate assessment criteria have been decided, they should be justified, and both the 

criteria themselves and their justification made available to all students. Once students are assured that group work 

is beneficial, and that they will be judged according to their individual efforts, much of the initial antipathy will 

dissipate. 

Problem #2: the selection of the groups 

Selection of groups tends to be easier in an online environment. Two problems common in the face-to-face 

environment are either non-existent, or greatly reduced: the tendency for students to want to be in a group with 

friends (and to feel aggrieved if they are not), and the difficulty of arranging suitable times when all group members 

can meet outside of scheduled sessions. 

How large should each group be? There is no standard answer here that fits all circumstances. Johnson and Johnson 

(1987) and Kagan (1998) suggest that teams of four work well in a face-to-face setting, while Bean (1996, p160) 

suggests groups of five or six work best. However, arguments based on group dynamics are less applicable in an 

asynchronous online environment, where both small and large groups can work well, depending upon the context, 

and the size and complexity of the group task. 

How should the membership of each group be determined? There are several solutions here, but letting students 

choose their own groups is not usually one of them. The method presents too many complications in the online 

environment. 

Solution #2.1: Select at random. 

Since the students are online, and can, presumably, communicate via email, problems of geographic location shrink 

into unimportance, and, if communication is asynchronous, so too does the problem of arranging meeting times. So 

group membership based on a random selection is likely to be prone to far fewer difficulties than would be the case 

in a face-to-face environment. And a pseudo-random selection (free from even unconscious instructor influence), 

based on a digit in the student number, or the month of birth, or some other criteria, is easy to implement. 

The down side to such a solution is obvious: as Burdett (2003: p178) comments: 

‘…it is likely that groups will be formed with little consideration given to personality, life experience, 

ability or aptitude, so that a successful mixture of individuals is more likely to be achieved by happy 

accident rather than design.’ 

In many cases, however, a random selection may suffice, and may indeed prove to be as effective as some more 

contrived method. 

259

Solution 2.2: Deliberately select heterogeneous groups. 

Left to select on their own, students will naturally choose to be with friends, who, more likely than not, will be from 

similar backgrounds. It can be very beneficial for the instructor choosing groups to attempt to do exactly the 

opposite – that is, to mix students from a range of ages, genders, and cultural backgrounds. This can improve a range 

of generic skills, including the ability to communicate effectively, to understand others’ points of view, and to be 

understanding of other cultures and backgrounds. 

There is some evidence that heterogeneous groups can be advantageous, because of the different perspectives 

brought to the group (Kagan, 1997; Johnson & Johnson, 1989). Thus, where possible, it would seem advisable to 

consider each student’s previous academic background and work experience as important factors. In fact, the 

authors’ own experiences would suggest that it is often considerably easier to form diverse groups when the majority 

of students are studying online, rather than face-to-face. 

Problem #3: a lack of essential group-work skills 

‘Simply placing students in groups and telling them to work together does not in and of itself result in cooperative 

efforts. There are many ways in which group efforts can go wrong.’ So say Johnson and Johnson (1994: p57), and 

of course they are quite right. Educators need ‘to foster students’ group skills over time, building more complex 

group activities as students become more familiar with the group context.’ (Teaching & Assessment Network, 1999: 

p5). Burdett (2003: p179) stresses the point that: ‘group work can be hard work emotionally and intellectually; and 

that this fact is sometimes overlooked by group work advocates and practitioners’. 

In situations where students who have not previously been introduced to group work, and lack the necessary skills, 

any instructor who uses group work as a major component, and does not prepare the students appropriately, is almost 

inevitably condemning the students to a traumatic and probably unproductive experience. This is certainly one of the 

major reasons why some instructors choose to revert to more traditional methods. 

Solution #3.1: introduce new courses. 

Potentially the most powerful way to ensure that students are both enthusiastic and appropriately prepared for group 

learning is the introduction of a core course to cover the requisite skills. This has the dual effect of instilling in 

students the idea that the university regards such skills as of significant importance, and of providing a sound 

preparation for future courses. Of all of the solutions presented in this paper, this is the only one that may lie outside 

the purview of the instructor, since it requires a change to the overall program, rather than to the individual course. 

As such, it may only be possible to implement this solution with the agreement of program administrators and other 

educators. 

The Teaching & Assessment Network (1999: p4) states the ‘…need to ease the introduction of group work into an 

otherwise “traditional” degree framework, fit group work into the semester system and to train academic staff in 

their understanding of group skills and theory’. The importance of these three aspects cannot be overstated - in 

particular, the need to provide not only the students, but also the instructors, with the necessary group work skills. 

Ideally, a short course for staff should be run on a regular basis, and an introductory core course for students, in the 

first year of the program, should be introduced. Both staff and student courses should cover key generic skills, of 

which group work, team-building, and effective communication would form an integral part (other parts of the 

course could cover topics such as computer literacy, presentation skills, email etiquette, proper referencing, etc). 

This solution enables any courses within the program to utilise group learning, confident that the key requirements 

and skills will already be familiar to the students prior to the commencement of the course. Similarly, students will 

be aware that group work may form part of the standard learning process, and are likely to approach such courses 

with far less trepidation than might otherwise be the case. 

260

Solution #3.2: cover the skills required at the beginning of the course. 

In cases where students participate in group work without any prior formal training in group skills, a minimum of 

two weeks at the start of the course should be devoted entirely to the core advantages and benefits of group learning, 

and the skills required. This may seem difficult – perhaps impossible – to many educators convinced that they have 

to “cover the content”, and that they “cannot afford to waste two weeks”. However, preparation of students in this 

way is an essential prerequisite for successful group work, and “covering the content” somehow assumes less 

importance when one steps out of the more usual lecture / seminar / tutorial mode. 

Amongst the skills that should be stressed in these sessions are group facilitation, effective online communication, 

‘netiquette’, and responsibilities to other group members. Excellent discussions of the processes involved in a 

successful group formation phase can be found in Kagan (1997) and Daradoumis and Xhafa (2005). 

Problem #4: the free-rider 

The free-rider effect (Kerr and Bruun , 1983) is probably the most commonly cited disadvantage of group work; that 

is, when one or more students in the group does little or no work, thereby contributing almost nothing to the well 

being of the group, and consequently decreasing the group’s ability to perform to their potential. In many cases, this 

may multiply into additional unwanted effects: first, of gaining unwarranted marks for the free-rider; second, of 

damaging the morale of the other members of the group; and third, of lowering the reputation of the educator and the 

institution for fair dealing and justice in assessment. 

An example of this attitude is illustrated by Burdett (2003: p178), quoting a University of South Australia graduate 

student: 

‘I acknowledge the reasons for including group work as a component of a university course; however 

due to the nature of groups, it usually falls to one or two individuals to do the bulk of the work. As a 

student motivated to achieve the best results of which I am capable, I find it frustrating that not only 

do other students get a free ride so to speak, but that through being forced to work in groups, the task 

becomes more difficult than it would have been if done alone.’ (University of South Australia, 2001) 

Levin (2002b: p3) states that the educator ‘may be the last person to know that there are students who consider that 

there is a free-rider’ hiding in their group. Students are ‘likely to feel that the issue is one that they should deal with 

themselves, and … be reluctant to tell tales on a fellow student’. It is appropriate then that educators pride an 

environment in which students involved in group work indicate the responsibilities they will be undertaking within 

the group, as a means of maintaining the integrity of the group and as a way to lessen the free-rider effect. 

Solution #4.1: use pressure from the instructor. 

The instructor should make potential free-riders within the group aware that they will lose marks – and indeed run 

the risk of failing the course – if they do not contribute. It is not sufficient – nor indeed fair – to impose rulings to 

this effect only at the end. It is essential that assessment rules be made explicit prior to the commencement of all 

group work. 

Unfortunately, if this is the only method used, constant monitoring by the instructor is essential. This can be timeconsuming 

and stressful at best, particularly in an online environment, and other solutions are generally preferable. 

Solution #4.2: use peer pressure openly and unashamedly. 

The best monitors of the quantity and quality of any single student’s contributions are the other students within the 

group. As such, students who seem to be free-riding should be encouraged by the other members of their group to 

‘pull their weight’. Such reminders should occur on a regular basis throughout the entire group learning process – not 

261

just near a deadline, when it may be too late. It is therefore a primary responsibility of the instructor to ensure that all 

group members are aware right from the start of their own responsibilities in this area. 

Many educators are already doing this to good effect. For example, Chin and Overton (2005: Pp 2-3) state in their 

primer that: 

‘Depending on the assessment tools employed students can both receive and provide feedback to their 

peers. This process helps students gain a better appreciation of the skills being developed and how to 

work effectively as a group. For example, peer assessment of a presentation can improve student 

understanding if they have to assess their peers on the same criteria with which they will be assessed.’ 

Such assessment does not have to include any element of marking. However, the best solutions do just that. 

Solution #4.3: employ a marking scheme that penalises free-riders 

The most appropriate and effective antidote to the free-rider problem is to ensure that all students are well 

acquainted, at the commencement of group work, with the marking scheme to be employed, and that such a scheme 

is clearly seen to penalise free-riders. 

The unfortunately-common scheme of giving equal marks to all group members is not recommended, for such a 

scheme almost invariably invites free-riders to take advantage of the other members of the group. Why would the 

savvy and more industrious student spend time on this subject, since work by other students will get the marks, and 

their time can be more profitably spent on other subjects? Therefore, a marking scheme needs to be employed that 

provides different marks to group members based upon their individual contributions. 

One such method is to build into the assessment process an element of peer and/or self-assessment, thereby giving 

students the ability to demonstrate independent value within the group confines. Watkins and Daly (2003: p.9) have 

suggested a seemingly effective assessment method whereby the group participants are awarded bonus points by 

their peers, in the hope that this 

‘…may reduce social loafing through the ‘group evaluation’ effect and they may reduce free riding by 

enabling some equitable allocation of outcomes to individual input.’ 

Relying as it does on differential individual assessment, this solution is covered more fully in problem #7. 

Problem #5: possible inequalities of student abilities 

Many researchers have expressed the hope that the online environment would, of itself, produce more equal levels of 

group participation than might be expected in a face-to-face environment (Harasim, 1993; Harasim et al, 1995; 

Sproull & Kiesler, 1991). This hope has not always been well-founded, however, with certain individuals or groups 

often dominating discussion (eg Herring, 1993). 

Winkworth and Maloney (2002) state that a fundamental dilemma in groups can be the need to temper the individual 

students’ needs with those of all the students in the group. In successful groups, individual students may need to 

sacrifice some aspects of their individuality for the benefits of learning in a group (Roschelle & Teasley, 1995). One 

of the supervisor's tasks is to monitor the groups to try to ensure that the strengths (and not the weaknesses) of 

individual students’ abilities are activated, while trying to ensure the success of the group as a whole. 

There is always the possibility that the most able student(s) within a group may fall victim to what has become 

known as the sucker effect (Kerr, 1983), which in many ways may be the reverse of the free-rider effect. The sucker 

in the group is the student who is perceived by other members of the group to be the most capable, and is therefore 

left to carry the bulk of the workload. 

262

It should be noted at the outset that the sucker effect is not all bad. It can result in weaker students learning more 

effectively, and perhaps going on to be suckers themselves in later groups. The situation that must be guarded against 

is one where the sucker does all the work, and is not rewarded appropriately for it. Also, of course, it is possible that 

the other students within the group rely so much on the sucker to do the work that they fail to learn anything at all. 

In certain circumstances, the free-rider and sucker effects can feed on each other. Ruël et.al. (2003: p.3) aptly stated 

that ‘…due to a feeling of being exploited by free-riders, one also reduces one’s own effort, because he or she does 

not want to be seen as a sucker who does all the work for his or her co-students’, and noted that there are several 

conditions which will create the sucker effect, including ‘…the type of task to be performed, the number of students 

within a team (group size), the type of performance and reward (on an individual or a group basis), the 

identifiability of the individual contribution and certain group characteristics.’ (Ruel et.al., 2003: p.3) 

The good news is that the sucker effect is fairly easy to overcome. 

Solution #5.1: identify potential “suckers” in advance. 

It may be possible to identify potential “suckers” in advance – they will generally have proved themselves via past 

work to be very able students. If so, some confidential correspondence between instructor and “sucker” may be all 

that is required, stressing the benefits of group work, and that perhaps all within the group may benefit from 

participation, rather than being supplied with finished work by the “sucker”. 

Solution #5.2: employ an appropriate reward scheme. 

Just as there should be potential penalties for the free-rider, so there should be potential rewards for the sucker. If the 

rewards are sufficiently high, for example better marks, every group member will want to be a sucker, and the group 

may then out-perform expectations. If, despite best efforts, a clear “sucker” still emerges, the potential rewards 

should be arranged in order to be appropriate to efforts (eg Webb, 1994: p13). 

For example, Watkins and Daly (2003: Pp.10-12) discuss the use of bonus points for group members that combats 

both the free-rider effect and the sucker effect. Perhaps the fairest and most easily defensible method is to use an 

appropriate combination of self, peer, and group assessment techniques, where assigned marks are correlated with 

individual efforts, as judged by the members of the group themselves. The sucker will likely not object to being a 

sucker if he or she is adequately recognised by other members of the group, and their efforts appropriately rewarded, 

and other members are more than often happy with such an outcome. Successful methods such as these are 

described in detail by many chapter authors in (Roberts, 2005). 

Solution #5.3: use subgroups within groups where feasible 

Apart from suggesting the use of bonus points, Watkins and Daly (2003: Pp.10-12) also put forward the idea of using 

small subgroups as a method of creating a more equitable working situation. Such groups-within-groups generally 

make the free-rider work harder, and ensure that the sucker does not have to carry the entire group. Alternatively, 

instructors may find that it is easier to begin with a smaller group size; this small size will not automatically mean 

that there will be no free-rider or sucker effect, but it does mean it is easier to observe and intervene if necessary 

when it occurs. 

Problem #6: the withdrawal of group members 

Courses conducted online or by distance education notoriously suffer from higher than average attrition rates (eg 

Simonson, 2000), often because of feelings of isolation (Hara and Kling, 2000). In a more conventional learning 

environment, one where group work is not being used, the withdrawal of a student normally has little or no direct 

effect on the work or grade of other students. Of course, this may not be strictly true, since the student may be in an 

informal study group, or have developed a friendship with other students, etc; but students learn and are officially 

263

assessed on an individual basis. With group work, it is common for those students who remain in the group to feel 

disadvantaged if one or more of their members officially withdraws, or disappears from the group for whatever 

reason. 

Although probably the least-cited of the seven problems listed in the paper, this has the potential to be the most 

serious, since the student concerned may have been assigned some component vital to the success of the group as a 

whole. While the withdrawn student can be awarded zero, what should happen to the other members of the group? 

Solution #6.1: take no action. 

In circumstances where the group member drops out very early in the course, or does not play a vital role within the 

group, it may be appropriate to take little action other than a minor reassignment of roles, which could be managed 

by the instructor, or by the group members themselves, or a combination of both. In some instances, constant 

monitoring of the group may be enough to alert others to the possibility of such an occurrence, which can then lead 

to early intervention before the matter becomes crucial. 

Solution #6.2: use a multiplier on the group work. 

Despite effective monitoring, it often occurs that a student will withdraw at a vital stage of a groups work, and 

sometimes this may be completely unanticipated. This can happen for any number of reasons, some quite beyond the 

control of the student concerned. Personal circumstances change, accidents happen, etc. How can the other students 

within the group be treated fairly in such cases? Taking contributions so far into account, it may be possible for the 

instructor to grade each member of the group in the normal way, and then apply a multiplier to make up for the 

disruption. The size of the multiplier of course will be dependent upon the particular circumstances, how late the 

withdrawal occurred, and the degree to which the missed contribution played a role in the outcome of the group. 

Solution #6.3: use a multiplier on the remaining course work. 

In the most problematic cases, the instructor may decide that the effect of the withdrawal is such that the group 

cannot be assessed. In such cases, a multiplier can be used on the remaining assessment tasks, such as other group 

tasks, or individual components such as the end-of-semester examination. The imposition of extra assignment items 

to “make up” for the missed group work is generally not a recommended course of action, since this can be viewed 

as an added imposition on students who have had to cope with circumstances beyond their control, and have 

otherwise fulfilled all of the necessary requirements. 

Problem #7: the assessment of individuals within the groups 

The traditional view of assessment has always been something along the following lines: assessment is about 

grading. One or more instructors assess the work of the students, with the primary – and perhaps sole - aim of 

assigning fair and appropriate grades to each of the students at the end of the course. An alternative view, and one 

that has claimed a large number of adherents in recent years, is that assessment can and should play a vital part in the 

learning process itself (Bain, 2004; Roberts, 2005). No matter where one stands on this issue, however, at the end of 

the day, individual students must be assessed. How can this be done fairly if group work is used? 

Assigning group grades without attempting to distinguish between individual members of the group is both unfair 

and deleterious to the learning process, for many reasons which should be apparent from earlier discussion, and may 

in some circumstances even be illegal (Kagan, 1997; Millis and Cottell, 1998). 

Specifically talking about group work, Webb (1994) stated that the 

‘… purpose of assessment is to measure group productivity…’ 

264

ut then went on to stress that another purpose of assessment is 

‘… to measure students’ ability to interact, work, and collaborate with others and to function 

effectively as members of a team. Team effectiveness involves many dynamic processes including, for 

example, coordination, communication, conflict resolution, decision-making, problem solving, and 

negotiation.’ 

Several effective solutions may be employed to do exactly as Webb suggests, that is, to measure group productivity 

and to measure the individual students’ abilities within the group. Exactly which of the solutions is the most 

appropriate will depend upon the circumstances. 

Solution #7.1: use individual assessment. 

While the learning may take place in groups, it may still be appropriate to assess individually. For example, while 

skills may be built up by a series of group projects throughout the semester, the assessment of the student learning 

process may take place through individual tests or assignments placed throughout the semester, or via an end-ofsemester 

examination, or via a combination of these. 

This may be a perfectly valid solution in many cases. The down side is that some students may see little value in 

participating in the groups, if such work is not directly assessed. It is therefore up to the instructor to ensure that the 

assessment items, while being individual, nevertheless test the learning that has occurred within a group setting. 

This may not be easy. 

Solution #7.2: assess individual contributions. 

If the group work is to form the bulk of the assessment, the instructor may be in a position to assess the contributions 

of individual students throughout. This method may be employed in either the face-to-face or the online situation, but 

is perhaps most effective in the latter, since individual contributions can be stored and reflected upon before final 

grading takes place. 

Instructors may in addition require students to record their own contributions and reflect upon them, in the form of a 

diary or journal, or perhaps in the form of a more structured portfolio, to be submitted to the instructor at the end of 

the course. 

Solution #7.3: use self, peer, and group assessment techniques. 

Self, peer, and group assessment techniques can be extremely beneficial for both students and instructors in all forms 

of online collaborative learning. Students who learn in groups are generally very aware of their own, and others’, 

relative contributions to the group. This knowledge can be usefully employed during assessment. 

A number of strategies to determine individual grades based on peer reviews by other students are possible – see for 

example (Li, 2001). One technique is to have students within each group anonymously rate their fellow group 

members. For example, one scheme has the students log on at the end of each piece of group work, and 

anonymously rate their fellow members on a scale ranging from -1 to +3. A rating of -1 indicates the group member 

was actually deleterious to the group (that is, the group would have performed better had the student not been in the 

group); 0 indicates that the group member’s contributions were negligible or non-existent; +1 indicates a below 

average contribution; +2 indicates an average contribution; and +3 indicates an above average contribution. The 

mark for the performance of the group as a whole is then divided appropriately to each group member. A number of 

different formulae can be used for this, depending upon the requirements of the instructor. 

An alternative scheme uses a pie chart. Students are advised to divide up the pie according to their relative 

contributions to the group. Since this is done by all students anonymously and online, there is little fear of 

repercussions from aggrieved students. Experience has shown that this method generally works well, and is accepted 

265

– and even appreciated – by students. Once again, the instructor retains responsibility for final grades, but utilises 

the student’s recommendations when deciding how to reward individual contributions. 

A variety of other techniques utilising self, peer, or group assessment in an online learning environment can be found 

in (Roberts, 2005). Sample forms that may be used as rubrics for self, peer, and group assessment can be found in 

Barkley et al (2005). 

Conclusion 

This paper has attempted to help those considering the introduction of online group learning into their courses, by 

listing seven of the most common problems, and describing solutions for each. It may of course be argued that this is 

far from an exhaustive list, and that there are other potential problems of online group learning that have not been 

dealt with here. While the authors agree that is undoubtedly true, they are also of the belief that the benefits of 

online group learning are compelling, and they hope the solutions presented here will be sufficient to encourage 

other educators to take the risks, discover the benefits for themselves, and report the results, so that others may be 

likewise enthused. 

References 

Bain, K. (2004). What the Best College Teachers Do, Cambridge, MA: Harvard University Press. 

Barkley, E. F., Cross, K. P., & Major, C. H. (2005). Collaborative learning techniques, San Francisco: Jossey-Bass. 

Bean, J. C. (1996). The professor’s guide to integrating writing, critical thinking, and active learning in the 

classroom, San Francisco: Jossey-Bass. 

Bonwell, C., & Eison, J. (1991). Active Learning: Creating Excitement in the Classroom, retrieved October 15, 2007, 

from http://www.ntlf.com/html/lib/bib/91-9dig.htm. 

Burdett, J. (2003). Making Groups Work: University Students’ Perceptions. International Education Journal, 4 (3), 

retrieved October 15, 2007, from http://ehlt.flinders.edu.au/education/iej/articles/v4n3/Burdett/paper.pdf. 

Chin, P., & Overton, T. (2005). Assessing group work, retrieved October 15, 2007, from 

http://www.heacademy.ac.uk/assets/ps/documents/primers/primers/ps0083_assessing_group_work_mar_2005_1.pdf. 

Daradoumis, T., & Xhafa, F. (2005), Problems and Opportunities of Learning Together in a Virtual Learning 

Environment. In Roberts T.S. (Ed.), Computer-Supported Collaborative Learning in Higher Education, Hershey, 

PA: Idea Group Publishing, 218-233. 

Felder, R. M., & Brent, R. (1994). Cooperative learning in technical courses: procedures, pitfalls, and payoffs, 

retrieved October 15, 2007, from http://www.ncsu.edu/felder-public/Papers/Coopreport.html. 

Graham, C.R., & Misanchuk, M. (2004). Computer-Mediated Learning Groups: Benefits and Challenges to Using 

Groupwork in Online Learning Environments. In Roberts, T. (Ed.), Online Collaborative Learning: Theory and 

Practice, Hershey, PA: Information Science Publishing. 

Hara, N., & Kling, R. (2000). Students' distress with a web-based distance education course. Information, 

Communication and Society, 3 (4), 557-579. 

Harasim, L. (1993). Collaborating in Cyberspace: Using computer conferences as a group learning environment. 

Interactive Learning Environments, 3 (2), 119-130. 

Harasim, L., Hiltz, S. R., Teles, L., & Turoff, M. (1995). Learning Networks- a Field Guide to Teaching and 

Learning Online, Cambridge, MA: The MIT Press. 

266

Herring, S. (1993). Gender and democracy in computer-mediated communication. Electronic Journal of 

Communication, 3 (2), 1-17. 

Johnson, D.W., & Johnson, R.T. (1977), Learning Together and Alone; Cooperation, Competition and 

Individualization, Eaglewood Cliffs, NJ: Prentice-Hall. 

Johnson, D.W., & Johnson, R.T. (1984). Circles of Learning, Washington, DC: Assoc. Supervision and Curriculum 

Dev. As quoted by The Foundation Coalition (2005). Positive Interdependence, Individual Accountability, 

Promotive Interaction: Three Pillars of Cooperative Learning, retrieved October 15, 2007, from 

http://www.foundationcoalition.org/publications/brochures/acl_piiapi.pdf. 

Johnson, D.W., & Johnson, R.T. (1987). Creative conflict, Edina, MN: Interaction Book Company. 

Johnson, D. W., & Johnson, R.T. (1989). Cooperation and competition: Theory and research, Edina, MN: 

Interaction Book Company. 

Johnson, D. W., & Johnson, R. (1994). Leading the cooperative school (2 nd Ed.), Edina, MN: Interaction Book 

Company. 

Kagan, S. (1997). Cooperative learning, San Juan Capistrano, CA: Kagan Cooperative Learning. 

Kagan, S. (1998). Teams of four are magic! Cooperative Learning and College Teaching, 9 (1), 9. 

Kerr, N.L. (1983). Motivation losses in small groups: A social dilemma analysis. Personality and Social Psychology, 

45, 819-828. 

Kerr, N.L., & Bruun, S.E. (1983). Dispensability of member effort and group motivation losses: free rider effects. 

Journal of Personality and Social Psychology, 44, 78-94. 

Koschmann, T.D. (1999). Computer support for collaboration and learning. Journal of the Learning Sciences, 8, 495- 

498. 

Koschmann, T. (2001). Revisiting the paradigms of instructional technology. In G. Kennedy, M. Keppell, C. 

McNaught & T. Petrovic (Eds.), Proceedings of the 18 th Annual Conference of the Australian Society for Computers 

in Learning in Tertiary Education, Melbourne: The University of Melbourne, 15-22. 

Lave, J., & Wenger, E. (1991). Situated Learning: Legitimate Peripheral Participation, Cambridge, UK: Cambridge 


Levin, P. (2002a). Teamwork Tutoring: Helping Students Working On Group Projects To Develop Teamwork, 

retrieved October 15, 2007, from http://www.teamwork.ac.uk/MGS_teamwork_tutoring.PDF. 

Levin, P. (2002b). Running group projects: dealing with the free-rider Problem. Planet, 5, 7-8. 

Li, L (2001). Some Refinements on Peer Assessment of Group Projects. Assessment and Evaluation in Higher 

Education, 26 (1), 5-18. 

Lipponen, L. (2002). Exploring foundations for computer-supported collaborative learning. In G. Stahl (Ed.), 

Proceedings of the Computer-supported Collaborative Learning 2002 Conference, Hillsdale, NJ: Erlbaum, 72-81. 

Lipponen, L., Rahikainen, M., Hakkarainen, K., & Palonen, T. (2002). Effective participation and discourse through 

a computer network: Investigating elementary students' computer-supported interaction. Journal of Educational 

Computing Research, 27, 353-382. 

Millis, B. J., & Cottell, P. G., Jr. (1998). Cooperative learning for higher education faculty, Phoenix, AZ: The Oryx 

Press. 

267

Panitz, T., & Panitz, P. (1998). Encouraging the Use of Collaborative Teaching In Higher Education, retrieved 

October 15, 2007, from http://home.capecod.net/~tpanitz/tedsarticles/encouragingcl.htm. 

Roberts, T. S. (2004). Online Collaborative Learning in Higher Education, Hershey, PA: Information Science. 

Roberts, T. S. (2005). Self, Peer, and Group Assessment in E-Learning, Hershey, PA: Idea Group Publishing. 

Roschelle, J., & Teasley, S. (1995). The construction of shared knowledge in collaborative problem solving. In 

O'Malley, C.E., (Ed.), Computer Supported Collaborative Learning, Heidelberg: Springer, 69-97. 

Ruël, G. C., Bastiaans, N., & Nauta, A. (2003). Free-riding and team performance in project education, retrieved 

October 15, 2007, from http://som.eldoc.ub.rug.nl/FILES/reports/themeA/2003/03A42/03a42.pdf. 

Simonson, M (2000). Equivalency Theory and Distance Education, Tech Trends. 43 (5), 5-8. 

Slavin, R. (1987). Cooperative Learning: Student Teams, Washington: National Education Association. 

Sproull, L., & Kiesler, S. (1991). Connections: New ways of working in the networked organization, Cambridge, 

MA: MIT Press. 

Teaching & Assessment Network (1999). Developing and assessing students’ group work skills: Notes from the 

meeting of the Teaching & Assessment Network 16th December 1999, retrieved October 15, 2007, from 

http://www.le.ac.uk/teaching/tan/pdf/groups.pdf. 

Tinzmann, M. B., Jones, B. F., Fennimore, T. F., Bakker, J., Fine, C., & Pierce, J. (1990). What is the collaborative 

classroom, retrieved October 15, 2007, from http://www.arp.sprnet.org/Admin/supt/collab2.htm. 

University of South Australia (2001). Student experience survey 2000, Unpublished report, Adelaide: University of 

South Australia. As quoted in Burdett, J. (2003). Making Groups Work: University Students’ Perceptions. 

International Education Journal, 4 (3), retrieved October 15, 2007, from http://ehlt.flinders.edu.au/education/iej/ 

articles/v4n3/Burdett/paper.pdf. 

Vygotsky, L.S. (1978). Mind and Society: The Development of Higher Mental Processes, Cambridge, MA: Harvard 


Watkins, R., & Daly, V. (2003). Issues raised by an Approach to Group Work for Large Numbers. Paper presented 

at the BEST Conference, April 9-11, 2003, Brighton, UK. 

Webb, N. (1994). Group Collaboration in Assessment: Competing Objectives, Processes, and Outcomes, retrieved 

October 15, 2007, from http://www.eric.ed.gov/ERICDocs/data/ericdocs2sql/content_storage_01/0000019b/ 

80/13/69/9f.pdf. 

Winkworth, A., & Maloney, D. (2002). An exploration of apathy and enthusiasm in task-focused groups: 

implications for task design and supervisor intervention, retrieved October 15, 2007, from 

http://www.tedi.uq.edu.au/conferences/teach_conference00/papers/winkworth-maloney.html. 

268

Kwon, S. Y., & Cifuentes, L. (2007). Using Computers to Individually-generate vs. Collaboratively-generate Concept Maps. 


Using Computers to Individually-generate vs. Collaboratively-generate 

Concept Maps 

So Young Kwon 

The University of Texas Health Science Center at Houston School of Nursing, USA // Tel: +1 713 500 2069 // Fax: 

+1 713 500 0272 // soyoungkwon@gmail.com 

Lauren Cifuentes 

Department of Educational Psychology, Texas A&M University, USA // Tel: +1 979 845 7806 // laurenc@tam.edu 

ABSTRACT 

Five eighth grade science classes of students in at a middle school were assigned to three treatment groups: 

those who individually concept mapped, those who collaboratively concept mapped, and those who 

independently used their study time. The findings revealed that individually generating concept maps on 

computers during study time positively influenced science concept learning above and beyond independent use 

of study time, but that collaboratively generating concept maps did not. Students in both individual and 

collaborative concept mapping groups had positive attitudes toward concept mapping using Inspiration software. 

However, students in the collaborative concept mapping group did not like working in a group. This study 

contributes to the limited body of knowledge concerning the comparative effectiveness of individually and 

collaboratively-generating concept maps on computers for learning. 

Keywords 

Collaborative learning, Computer-based learning, Concept mapping, Meaningful learning, Science learning 

Introduction 

Within a constructivist framework, learning takes place as learners progressively differentiate concepts into more 

complex understandings and also reconcile abstract understanding with concepts acquired from experience. New 

knowledge is constructed when learners establish connections among knowledge learned, previous experiences, and 

the context in which they find themselves (Bransford, 2000; Daley, 2002; Jonassen, 1994). Chang, Sung, and Chen 

(2001) propose that concept mapping, a form of visualization, is a powerful learning strategy consistent with 

constructivist learning theory in that it is a study strategy that helps learners visualize interrelationships among 

concepts (Duffy, Lowyer, & Jonassen, 1991). 

Scientists agree with educators that visualization facilitates thought. J. H. Clark states that, “Visualization has been 

the cornerstone of scientific progress throughout history. Much of modern physics is the result of the superior 

abstract visualization abilities of a few brilliant men. …Virtually all comprehension in science, technology and even 

art calls on our ability to visualize. In fact, the ability to visualize is almost synonymous with understanding. We 

have all used the expression ‘I see’ to mean ‘I understand.’” (as cited in Earnshaw & Wiseman, 1992, p. v). The 

authors propose that because visualization is a factor in scientific understanding, visualization of concepts should be 

a focus of science curriculum. 

That concept mapping facilitates learning has been demonstrated in bodies of research on visual aids and 

visualization processes. Research on visual displays or visual representations as adjunct aids to text has demonstrated 

that they facilitate both recall and comprehension (Gobert & Clement, 1999; Mayer, 1989; Mayer & Gallini, 1990). 

Conceptual understanding can also be facilitated by requiring students to build meaningful and appropriate mental 

representations and concrete, visual representations of concepts being taught (Gobert & Clement, 1999, p. 40). While 

visually representing concepts, learners construct knowledge rather than absorb others’ representations of 

knowledge. Making visual representations is a natural cognitive activity which facilitates active synthesis of concepts 

and phenomena (Ajose, 1999). 






269

Concept Mapping 

Concept mapping by students is a common visualization method assigned by instructors in elementary and secondary 

schools as well as adult learning environments to provide students with access to their own visual representations of 

knowledge structures (Gaines & Shaw, 1995). According to Novak (1998), concept mapping is a process of 

organizing and representing concepts and their relationships in visual form. Concept mapping is one tool that can 

overtly engage students in meaningful learning processes. Further, concept mapping promotes meaningful learning 

and retention of knowledge for long periods of time and helps students negotiate meaning (Hyerle, 2000; Novak, 

1990). Concept mapping is a schematic device for representing interrelationships among a set of concept meanings 

embedded in a framework of propositions and two-dimensional, hierarchical, node-link diagrams that represent 

verbal, conceptual, or declarative knowledge in visual or graphic forms (Quinn, Mintzes, & Laws, 2003). 

The effectiveness of concept mapping for learning has been tested in several contexts across many content areas. For 

instance, using a Biology Achievement Test and the Zucherman Affect Adjective Checklist as measures, Jegede, 

Alaiyemola, and Okebukola (1990) found that concept mapping raised mean scores on achievement in biology (p 

=.00) and decreased male students’ anxiety levels (p= .01) in fifty-one tenth grade students. In 1990 Novak, the 

originator of the term "concept maps," reviewed investigations of the effect and impact of concept mapping. Such a 

summary revealed that the process promotes novel problem solving abilities and increases students’ positive attitudes 

toward the content being studied. 

In a meta-analysis of 18 concept mapping studies, Horton, MacConney, Gallo, Woods, Senn, and Hamelin, (1993) 

concluded that concept mapping on paper has positive effects on both knowledge attainment and attitudes for 

students at the elementary level, middle school level, high school level, and the college level. Concept mapping 

raised individual student achievement in the average study by .46 standard deviations (from the 50th to the 68th 

percentile). Concept mapping also improved student attitudes toward content being studied. 

Concept mapping is quickly learned and easily understood by students; the minimum use of text makes it easy to 

scan for a word, phrase, or general idea; and visual representation allows for development of a holistic understanding 

that words alone cannot convey (Plotnick, 1997). Also, concept mapping provides a useful organizational cue for 

retrieving information and concepts from memory by providing a representation of interrelationships among that 

information and those concepts. In teaching, concept maps can be assessed as representations of declarative and 

procedural knowledge in the science classroom (Rice, Ryan, & Samson, 1998; Ruiz-Primo & Shavelson, 1996). 

In a controlled study by Cifuentes and Hsieh (2003a, 2003b), college students who (a) visualized interrelationships 

among science concepts, (b) mapped relationships between new concepts learned and prior learning, and (c) created 

connections both visually and verbally in their hand written study notes performed significantly higher on a test than 

those students who did not show such interrelationships, relationships, and connections in their study notes (p=.02). 

The researchers concluded that visualization is effective as a metacognitive strategy for college level students. Due 

to the theoretical propositions and evidence reported above, theorists and researchers agree that students should be 

encouraged to generate concept maps during study time. 

Computer-supported Visual Learning 

Incorporating visual thinking tools into the teaching and learning process opens up new avenues for constructivist 

learning (Anderson-Inman, 1996). Concept mapping can be directly and easily supported by personal computers and 

computer software (Anderson-Inman & Ditson, 1999; Anderson-Inman & Zeitz, 1993; Fischer, 1990; Royer & 

Royer, 2004). The use of computer-based visualization tools such as Inspiration and Visio enable learners to 

interrelate the ideas that they are studying in multidimensional networks of concepts, and to label and describe the 

relationships between those concepts (Jonassen, Carr, & Yueh, 1998; Jonassen, 2000). Anderson-Inman (1996) 

indicated that computer-based visualization makes the learning process more accessible to students, and it alleviates 

the frustration felt by students when constructing and revising concept maps using paper and pencil. 

Cifuentes and Hsieh (2004) explored the comparative effects of computer-based and paper-based visualization as 

study strategies for middle-schoolers' science concept learning. In their quasi-experimental study, although 

visualization on paper improved test scores for middle schoolers, scores on a test did not improve as a result of 

270

computer-based visualization during study. Qualitative findings indicated that students were quite unskilled at 

visualization and required training in both basic computer skills and computer-based visualization to successfully 

apply a computer-based visualization strategy. Students in that study also required more training in development of 

computer graphics to be effective visualizers on computers. 

Because of the finding that students require training in computer-based visualization, Hsieh and Cifuentes (2006) 

developed a seven-and-a-half hour visualization and computer skills workshop for preparing students to represent 

interrelationships among science concepts. In their controlled study, eighth grade students who used computer 

visualization as a study strategy outscored students who constructed visualizations on paper and those who did not 

construct visualizations at all during study time (p=.00). Hsieh and Cifuentes concluded that, given training in 

computer visualization, middle school students can generate visual representations that show interrelationships 

among concepts to both build and demonstrate their understanding. These findings provided evidence of the 

effectiveness of student-generated visualization using computers for the improvement of concept understanding. 

Royer and Royer (2004) also investigated the difference between hand drawn and computer generated concept 

mapping using Inspiration software on desktop computers with 9 th and 10 th grade biology classes. The group using 

the computer created more complex maps than the group that used paper and pencil. Also, students preferred using 

Inspiration to paper and pencil for concept mapping. Royer and Royer theorize that “computers enabled students 

to communicate more clearly, to add and revise concept maps more easily, and to discover relationships between 

sub-concepts more readily.” (p. 79) 

Collaboratively-generating Concept Maps 

According to Fischer, Bruhn, Grasel, and Mandl, (2002) collaborative processes can support learners’ scientific 

knowledge construction more effectively than independent processes. Lumpe and Staver (1995) demonstrated that 

collaboratively creating propositions using paper and pencil in small groups can have positive effects on student 

achievement. They compared collaborative conceptualizing of photosynthesis with individual conceptualizing of 

photosynthesis and found that high school students who collaborated out-performed those who worked 

independently on a comprehension test (p=.00). 

A visual representation technique such as concept mapping can be integrated into collaborative learning activities 

(Chiu, Wu, & Huang, 2000). During the process of collaboratively developing visualizations, the role of the student 

can evolve from being a passive learner to becoming an active, social learner. Students’ perceptions and 

representations of those perceptions are challenged during collaboration, and learning builds on what learners have 

already constructed in other contexts (Fischer, Bruhn, Grasel, & Mandl, 2002; Brandon & Holingshead, 1999). For 

instance, Novak and Gowin (1984) found that concept mapping provided for meaningful knowledge construction by 

providing a means for learners to communicate representations of their cognitive structures with other learners. Roth 

(1994) suggests that when students generate concept maps on paper in small groups, they are able to demonstrate 

what they know about a subject while listening, observing, and learning from others, resulting in the modification of 

their own meaningful understandings. 

In the only research study found investigating the comparative effects of individually vs. collaboratively generated 

concept maps, Brown (2003) compared test scores among students who collaboratively generated concept maps or 

individually generated concept maps on paper. A comparison of student comprehension of concepts showed that 

those students who collaboratively-generated concept maps on paper (M = 2. 34, SE = 0.56) outperformed students 

who individually generated concept maps on paper (M= 0.46, SE = 0.52) in high school biology. 

In summary, studies have investigated the effects of concept mapping on paper, concept mapping on computers, and 

concept mapping individually and collaboratively on paper. Such studies have shown that concept mapping 

positively affects students’ concept learning. When students have computer skills, computer-generated concept 

mapping also positively affects students’ learning of concepts beyond concept mapping on paper. In addition, the 

literature indicates that collaboratively generating concept maps on paper positively affects learning beyond 

individually generating concept maps. Therefore, the next logical step in the body of research on visualization is a 

comparison between computer-based individually-generated concept maps and computer-based collaborativelygenerated 

concept maps to determine which strategy is most appropriate. 

271

Research Questions 

In order to determine the comparative effects of individually vs. collaboratively-generating computer-based concept 

maps on eighth grade middle school science learning, the researchers administered treatments and compared scores 

on a subsequent comprehension test. Research questions were- (1) Did middle school students who collaboratively or 

individually generated computer-based concept maps perform better on a comprehension test than those who studied 

independently and did not generate computer-based concept maps? (2) Did middle school students who 

collaboratively generated computer-based concept maps perform better on a comprehension test than those who 

individually generated computer-based concept maps? (3) How did students’ attitudes toward generating concept 

maps during study time differ between those who individually-generated concept maps and those who 

collaboratively-generated concept map? And (4) What specific learning strategies were used in each group to prepare 

for the comprehension test and did they differ according to group? 

The researchers hypothesized that students who individually and collaboratively generated concept maps on 

computers would outscore those who did not, that students who collaboratively generated concept maps would 

outscore students who individually generated concept maps, that attitudes toward concept mapping would be positive 

across groups, and that students would apply different strategies across groups. 

Methodology 

Mixed methods were applied to answer the research questions. Using a quasi-experimental posttest-only-controlgroup 

design the relative effects of a computer-based individual concept mapping strategy, a computer-based 

collaborative concept mapping strategy, and a self-selected learning strategy on science concept learning were 

investigated. Posttest scores were compared across three treatment groups. Qualitative data were analyzed to 

describe attitudes toward concept mapping and study strategies employed across groups as well as to explain 

quantitative findings. 

Participants 

The potential participants were the entire eighth grade student body of a rural middle school in Texas (N=89). 

However, eleven of the students did not turn in consent forms, one was absent for part of the treatment, and three 

others were absent for testing. Therefore, 74 students (32 boys and 42 girls) in five eighth grade science classes 

participated in this study. All of those students took science from one teacher and all eighth grade students and 

science classes were assigned to that same teacher. The study was conducted in the natural school setting as part of 

the science curriculum with treatments randomly assigned to five classrooms taught by one teacher. This approach 

meant that student participants were not randomly selected and that treatment group sizes differed, two weaknesses 

of the study. 

The control group consisted of 12 classmates; the individual group consisted of 31 students from 2 different classes; 

and the collaborative group consisted of 31 students from 2 different classes. The ethnic distribution of the classes 

combined was 62% African American, 34% Hispanic, and 7% white. Over 84% of students were economically 

disadvantaged. Ethnicities and socio-economic status of students were equally distributed across the classes. 

Using mathematics and reading performance scores on the Texas Assessment of Knowledge and Skills to assure 

equivalence of student achievement across groups, the researchers randomly assigned the teacher’s classes to one of 

the three experimental groups. Those groups were- control, computer-based individual concept mapping strategy, 

and computer-based collaborative concept mapping strategy. Chi-square results indicated that no significant 

difference existed among the control, individual, and collaborative groups on their prior math and reading 

performance scores on the standardized Texas Assessment of Knowledge and Skills (X 2 = 1.13, p = .57 and X 2 = .30, 

p = .86 respectively). 

To determine whether the groups differed in their knowledge of the four science topics to be used in the experiment: 

“Tools of Modern Astronomy,” “Characteristics of Stars,” “Lives of Stars,” and “Star Systems and Galaxies,” 

students were asked to report the extent to which they had been previously exposed to the information presented in 

272

the science essays that they studied during the four day experiment. Four Pearson Chi-Square tests were conducted 

for the comparison among three groups. The Chi-square results indicated that there was no significant difference in 

group knowledge of the topics among the students across the four topics (X 2 = 4.82, p = .57; X 2 = 5.92, p = .43; X 2 = 

3.93, p = .69; and X 2 = 4.09, p = .67). 

In addition, Chi-Square tests were used to investigate whether the three groups were different from each other in 

their frequency of accessing computers at school and at home, in the number of computer courses taken in the past, 

in the amount of the time spent each time using a computer in school and at home, and in the frequency of using 

computer tools to support various learning tasks, such as word processing, E-mail, Internet, games, spreadsheets, 

presentations, graphics, and webpage development. According to their self-report, students in all three groups were 

not different regarding previous experiences using computers (see Table 1). 

Table 1. Pearson Chi-Square Group Differences in Computer Use Survey 

Topic Pearson Chi-Square Df Significance (2-sided) 

Use computers at school 2.22 4 .70 

Time spent at school computers 3.85 6 .70 

Numbers of computer courses 3.09 4 .54 

Frequency –use of computer at home 5.47 4 .24 

Time spent at home computers 2.78 4 .60 

Frequency – Create computer graphics 4.61 8 .80 

Frequency – Word 2.19 4 .70 

Frequency – Internet 2.72 4 .61 

Frequency – E-mail 6.13 8 .63 

Frequency – Chatting 5.89 6 .44 

Frequency – Games 3.47 6 .75 

Frequency – Spreadsheets 2.14 6 .91 

Frequency – Presentations 5.20 8 .74 

Frequency – Programming .65 2 .72 

Frequency – Webpage development 2.34 2 .31 

*p < .05. 

Design 

The independent variable, group, had three levels: a control group consisting of one class of twelve students who 

were not trained in concept mapping, an experimental group consisting of two classes totaling thirty-one students 

trained to individually generate concept maps on computers, and an experimental group consisting of two classes 

totaling thirty-one students trained to collaboratively generate concept maps on computers. The dependent variable 

was science concept learning as demonstrated by comprehension test scores. 

Computer-based Concept-Mapping Workshop 

Prior to studying science concepts, both groups that created concept maps on computers attended a three day 

workshop on computer-based concept mapping. The computer-based concept mapping workshop lasted fifty minutes 

each of three days. The workshop had the same content, materials, and processes for each experimental group except 

that students in the collaborative group were informed in the last five minutes of the last day of the workshop that 

they would be concept mapping collaboratively the following day. Science topics explored during the computerbased 

concept mapping workshops were “The eye – An organ system,” “Light waves and lenses,” and “Wave 

behavior.” The science content of the workshops were carefully selected to assure that content did not include 

concepts to be covered in the experimental studying materials. The first day of the workshop the teacher trained 

students focusing on how to identify and visualize expository text with sequential structures using Inspiration. On the 

second day of workshop, the teacher trained students to identify and visualize expository text with categorical 

structures. For the third day of the workshop, the same teacher trained students to identify and visualize expository 

text with compare-contrast structures. The control group spent the same amount of time as the experimental groups 

273

with their teacher but rather than learning how to create concept maps, they watched a video about the upcoming 

science fair. 

Procedures 

Prior to conducting the study, one of the researchers spent approximately one hour training the teacher in how to use 

Inspiration to create concept maps and then another hour training the teacher on how to deliver the Computerbased 

Concept-mapping Workshop to the students. Led by their teacher, students in the individual and collaborative 

experimental groups first spent three days learning how to develop concept maps on computers using Inspiration 

in the Computer-based Concept-mapping Workshop. Every student participant had a computer account and logon ID 

for the school computer laboratory so that the teacher was able to trace student work and outcomes on an 

administrator’s server. 

After three days of the workshop, the control, individual, and collaborative groups were given the same science 

essays to study in the classroom for four days. The only difference among groups was that the control group 

followed their own learning strategy to study the concepts. Computers were not available to them. In the individual 

group, students worked independently to use their learned computer-based concept mapping skills to show 

interrelationships among concepts during their study time. Each student worked alone at a computer. 

However, in the collaborative group, students were required to study together within groups of three per computer 

using their learned computer-based concept mapping skills to collaboratively create concept maps that showed 

interrelationships among concepts during their study time in the classroom. Each of the three group members was 

assigned to be either the group’s leader, reporter, or monitor. The role of the leader was to encourage group input and 

use the mouse to create the concept maps according to group input. The role of the monitor was to provide input 

regarding creation of bubbles and links. The role of the reporter was to print out group concept maps and summarize 

group work during study time. Roles were rotated daily so that each group member could fulfill each role. Students 

had not had prior training in how to work in such groups nor did they receive training as part of the experiment. The 

same teacher for all groups implemented instructional procedures during the four day experimental period. When 

students from any group asked for help and information, the teacher gave feedback equally to the students. 

The experimental procedure for all three groups followed three steps each of the four experimental days: First, as 

was typical in the classroom when students studied concepts in their text, the teachers gave ten minutes of instruction 

for each group. Second, after the teacher’s instruction, the students in the control group studied individually to 

prepare for the comprehension test. The control group students followed their own learning strategy such as 

highlighting, memorization, or taking notes to prepare for their test for thirty minutes. Students in the two 

experimental groups’ created concept maps using computers for thirty minutes. 

Students in the individual group created concept maps and studied independently using computers. Students in the 

collaborative group, however, created concept maps together using computers. The experimental groups’ students 

saved their files on their computer-server and printed out their concept maps to use them during their study time. 

Finally, all three groups of students turned in their study notes and concept maps prior to taking a test. 

Materials and Instruments 

The study essays were selected by the classroom teacher from the Prentice Hall Science textbook for eighth grade 

that was adopted by the school district (Padilla, Miaoulis, & Cyr, 2002). The contents of the four essays to be studied 

by students were validated by a subject matter expert and the teacher established that they met the state curriculum 

and that the students had not been exposed to those essays or the topics of those essays in school prior to the study. 

The four short essays, “Tools of Modern Astronomy,” “Characteristics of Stars,” “Lives of Stars,” and “Star Systems 

and Galaxies” were each one page long and consisted of expository text without illustrations or graphics. Students 

were given 50 minutes to study each essay. 

The comprehension test consisted of 40 computer-based multiple-choice items from the Prentice Hall test bank that 

was provided with the eighth grade textbook adopted by the participating school district. The multiple choice 

comprehension test items were selected and validated by both the teacher and researchers as appropriate for this 

274

study. Items were criterion referenced to concepts in the essays that students studied during their experimental study 

time. Ten item multiple choice comprehension tests were administered on each of the four days for ten minutes after 

treatment. Scores were totaled for each student to provide the 40 item total. For the purpose of scoring students’ 

responses to the comprehension test items, one point was given for a correct answer, and no credit was given for 

incorrect or unanswered questions. Internal consistency was established at .82 (coefficient alpha) for the 

comprehension test. 

All participants were asked to fill out a Learning Strategy Questionnaire in the last minutes of the fourth day. The 

Learning Strategy Questionnaire was a student self-report instrument developed by the researchers. Students in all 

three groups were asked to describe the steps that they took to prepare for the test. Students in both experimental 

groups were asked to explain how they felt about making concept maps that showed interrelationships among 

concepts during study time and to discuss how making concept maps helped them learn content. The individuallygenerated 

concept mapping group answered the following other questions: When you created concept maps on a 

computer during study time, do you think that working by yourself helped you learn the content better than if you 

had worked with others? Why or why not? The collaboratively-generated concept mapping group answered the 

following questions: When you created concept maps on a computer during study time, do you think that working 

with others helped you learn the content better than if you had worked by yourself? Why or why not? 

Data Sources and Analyses 

The four data sources included: (a) comprehension test scores, (b) student responses on the Learning Strategy 

Questionnaire, (c) students’ study notes, and (d) the video recording of classroom activities. Comprehension test 

scores were analyzed quantitatively. Students’ study notes, the video recording, and responses on the Learning 

Strategy Questionnaire were analyzed qualitatively to explain quantitative results and to provide insight into 

participants’ attitudes and study strategies. 

A one-way analysis of variance (ANOVA) using “treatment” as the independent variable and “comprehension test 

scores” as the dependent variable was administered among the control, individual, and collaborative groups. The 

researchers summarized student responses to the Learning Strategy Questionnaire and triangulated those self-reports 

with researchers' and teachers' observation and students' study notes. The researchers examined students' study 

notes and identified strategies employed. Content analyses approaches as described by Emerson, Fretz, and Shaw 

(1995) were applied to the researcher's journal entries, the video recording during study time, and students' response 

to the Learning Strategy Questionnaire. For focused coding analyses, the researcher independently compiled and 

numbered the contents according to the categories that emerged (Merriam, 1998). To analyze the video recording, 

the researcher watched the video several times and independently identified categories of student behaviors. 

Results 

Results of data analyses provided answers to the research questions regarding the effects of individually generated 

computer-based concept mapping, and collaboratively generated computer-based concept mapping. ANOVA 

(Analysis of Variance) revealed that means differed across groups. A .05 level was used for determining 

significance. Levene’s Test of Equality of Error Variances was applied and groups were found to be homogenous. 

Descriptive statistics for the experimental groups’ were respectively: individual group, n = 31, mean = 26.29, SD = 

4.49; collaborative group, n = 31, mean = 23.19, SD = 6.20; and the control group, n=12, mean = 19.67, SD = 7.11. 

The one-way ANOVA results indicated that a significant difference existed among the individual, collaborative, and 

control, groups on the mean scores of the comprehension posttest, F =6.25 (p < .05 level) as seen in Table 2. 

Table 2. One Way ANOVA Summery Table 

Source Df Mean Square F Significance 

Group 2 203.81 6.25 .00* 

Error 71 2315.89 

*p < .05. 

275

Tukey’s HSD post hoc test revealed that the group that generated concept maps individually significantly outscored 

the group that did not generate concept maps (control group), while the group that generated concepts maps 

collaboratively did not differ significantly in its performance when compared to the other groups (see Table 3). The 

effect size between the control and individual groups was 1.11. The effect size between the control and the 

collaborative groups was 0.53. The effect size between the individual and collaborative groups was 0.57. 

Table 3. Tukey HSD Post Hoc Test Results 

Tukey HSD 

(I) Group (J) Group Mean Difference Significance Effect Size/Cohen’s d 

Control Individual 6.62* .00* 1.11 

Control Collaborative 3.53 .17 0.53 

Individual Collaborative 

*p < .05. 

3.10 .09 0.57 

Therefore, our first research hypothesis that middle school students who collaboratively or individually generate 

computer-based concept maps perform better on a comprehension test than those who do not generate computerbased 

concept maps is only partially accepted. Although students who individually generated concept maps scored 

higher than the control group, students who collaboratively generated concept maps did not score significantly higher 

than the control group on the comprehension test. Also, there was no significant difference between the groups of 

students who individually and collaboratively generated concept maps. Therefore, the second hypothesis that middle 

school students who collaboratively generated computer-based concept maps performed better on a comprehension 

test than those who individually generated computer-based concept maps is rejected. Qualitative data provided 

insight regarding quantitative findings, students’ attitudes toward concept mapping, and the strategies that students 

chose to employ while studying. 

Table 4. Survey Result of Learning Strategy during Study Time 

Category Control Group Individual Group Collaborative Group 

Feeling about None 85% say helpful and fun. Some 87% say helpful and fun. Some 

*CM 

students think better than students think better than 

worksheet on paper or regular worksheet on paper or regular 

class activity. 15 % No class activity.13 % No 

response 

response 

Sense that CM None 87% think concept maps help 97% think concept maps help 

helps with 

with learning the science with learning the science 

content learning. 

content. 13 % No response content. 3 % No response 

Opinion of 

working with 

group 

Opinion of 

working 

individually 

Steps taken to 

prepare for test 

Previous 

exposure to 

concept mapping 

*CM = concept mapping 

None None 50% think working with a 

group is helpful and useful. 40 

% did not like group work and 

preferred to work alone. 10 % 

None 70% think that studying by 

themselves is helpful and 

useful. 19% preferred to work 

Read the whole handout 

and some students took 

notes and underlined parts. 

in a group. 11 % No response 

Generated and studied their 

concept maps and tried to 

understand relationships 

among the bubbles. 

None 12% had created concept maps 

before. 100% did not know 

Inspiration program before 

No response 

None 

Generated and studied their 

concept maps and tried to 

understand relationships 

among the bubbles. 

9% had created concept maps 

before. 100% did not know 

Inspiration program before 

276

Attitudes toward concept mapping for science concept learning were positive whether the students were working 

individually or collaboratively (see Table 4). Generating concept maps using the computer program, Inspiration, 

provided students with a useful learning strategy and a positive experience. An interesting observation by both the 

teacher and the researchers was that the students in the individual group were more positively engaged in their 

studying than were the students who studied collaboratively. The teacher and researchers observed students in the 

collaborative groups spending excessive time competing for control of the mouse and students complained vocally 

about having to share the keyboard and work collaboratively. Only 50% of students in the collaborative group 

thought that working with peers was helpful for studying and learning science concepts. Students’ negative attitudes 

toward collaboration may have influenced their comprehension of concepts being studied and might explain why the 

students in the individual group performed better than the students in the collaborative group. 

The study strategy chosen by students in the control group when they prepared for the test was to simply read the 

handout, while the students in the two experimental groups’ created and studied the relationships between bubbles on 

their concept maps and links that they created during study time. Students who created concept maps expressed that 

creating those maps and studying relationships between bubbles and links were quite helpful and fun for learning 

science. 

Discussion and conclusions 

The findings provide further evidence that individually-generating concept maps during study time positively 

influences science concept learning and that computer-based concept mapping can be facilitative. But findings do not 

support the assumption that collaborative learning is more effective than learning individually. Students enjoyed 

Inspiration software which supported their construction of concept maps for science learning and helped them 

capture their quickly evolving ideas and organize them for meaningful learning. These findings provide evidence that 

constructivist learning theory is correct regarding learners’ needs to organize and represent concepts visually and 

explore interrelationships among concepts. However, in this case, social construction of meaning using concept maps 

was no more effective than application of a self-selected study strategy. 

The Positive Effect of Concept Mapping on Learning 

These findings replicate previous research results (Cifuentes & Hsieh, 2003a, b; Cifuentes & Hsieh, 2004, Hsieh & 

Cifuentes, 2003; Hsieh & Cifuentes, 2006). It extends the research by providing evidence that individuallygenerating 

concept maps on computers is more effective than either independent, unguided study, or collaborativelygenerating 

concept maps. However, the findings do not support Fischer, et al’s (2002) assumption that collaborative 

knowledge construction is more effective than individual knowledge construction. Qualitative findings suggest that 

the reason that students in the collaborative group did not score significantly higher than the control group on 

achievement might have been lack of a disciplined, supportive collaborative working environment. 

Cifuentes and Hsieh (2004) previously demonstrated that distraction of computers and software and the difficulty of 

visualization can contribute to lack of the effectiveness of computer-based visualization. Students in the school 

setting of this study were not motivated to collaborate with each other and were distracted by each other, the 

computers, and the software. Most of the participants did not have computers at home and the school district had 

limited technical facilities. The participating students learned concept mapping using computers for the first time in 

the context of this study. In addition, according to their own self-report, most students had had few opportunities to 

develop collaborative learning skills in their young school careers. With computer skills, concept mapping, and 

collaboration all new to the students, the combined tasks challenged them. To be effective, all three components of 

the experiment required sophistication on the part of learners. 

Perhaps more experienced learners would produce a different result. Findings indicate that teachers should train their 

students in computer-based concept mapping and facilitate adoption of concept-mapping as an independent study 

strategy. Deciding whether to adopt a computer-based individual concept mapping strategy or a computer-based 

collaborative concept mapping strategy might be based upon characteristics of the learners and the learning context. 

For example, a teacher should ask the following questions prior to implementing concept mapping: Do students feel 

comfortable and competent working on computers during class time? Do students already know how to work 

277

collaboratively during study time? Do students know how to work collaboratively on computers? If the answer to 

any of the questions is “no,” as was the case in this study, then teachers should only recommend such a strategy after 

students have been sufficiently trained on computers, on collaboration, and on computer-based collaboration. Until 

that time, students should be encouraged to develop concept maps individually. If the answer to these questions is 

“yes,” then the teacher might consider encouraging students to generate concept maps collaboratively. 

Recommendations for Future Study 

Study limitations are that generalizations to populations beyond the sample of this study should be made 

conservatively because a nonrandomized, quasi-experimental design was used, a small number of students 

participated, and groups were of unequal size. Therefore, it is recommended that these findings be replicated in a 

study with a larger group of students who are randomly selected for placement in equal-sized groups. However, we 

acknowledge that this is quite difficult in the naturalistic classroom environment. 

Chiu, Wu, & Huang, (2000) have found that when students have computer skills and collaboration skills, they can 

work together effectively on computers. Therefore, the researchers suggest that this study be replicated in another 

school context, where students have technical skills and support, the atmosphere is conducive to collaboration, and 

students have a history of collaborative experience in school. They predict that under these circumstances, the 

outcomes of collaboratively-generated concept mapping may be more positive. Researchers might investigate 

whether individually or collaboratively computer-generated concept maps differentially affect learners with specific 

characteristics such as those listed above. 

Given that participants in this study were distracted by members of their group, both the classroom teacher of those 

students and the researchers think that an investigation comparing collaborative groups versus collaborative pairs is 

of interest. Collaborative pairs might be less distracting than collaborative groups. Further studies might be 

conducted to compare the effect of individually and collaboratively generating concept maps on the quality of those 

maps. 

Possible qualitative factors might include— propositions, hierarchical relationships among sub-concepts, cross links, 

and examples (Novak & Gowin, 1984). In order to conduct a study using quality of concept maps as a dependent 

variable, training in future studies should specifically prepare study participants to generate concept maps that 

include quality factors identified in concept mapping literature. 

References 

Ajose, S.A. (1999). Discussant’s comments: On the role of visual representations in the learning of mathematics. 

Paper presented at the annual meeting of the North American Chapter of the International Group for the Psychology 

of Mathematics Education, October 23-26, 1999, Morelos, Mexico. 

Anderson-Inman, L. (1996). Computer-assisted outlining: Information organization made easy. Journal of 

Adolescent and Adult Literacy, 39 (4), 316-320. 

Anderson-Inman, L., & Ditson, L. (1999). Computer-based concept mapping: A tool for negotiating meaning. 

Learning and Leading with Technology, 26 (8), 6-13. 

Anderson-Inman, L., & Zeitz, L. (1993). Computer-based concept mapping: Active studying for active learners. The 

Computer Teacher, 21 (1), 6-8, 10-11. 

Brandon, D. P., & Holingshead, A. B. (1999). Collaborative learning and computer-supported groups. 

Communication Education, 48 (2), 109-126. 

Bransford, J. (2000). How people learn: Brain, mind, experience, and school, Washington, DC: National Academy 

of Sciences. 

278

Brown, D.S. (2003). High School biology: A group approach to concept mapping. The American Biology Teacher, 

65 (3), 192-197. 

Chang, K.E., Sung, Y.T., & Chen, I.D. (2001). Learning through computer-based concept mapping with scaffolding 

aid. The Journal of Computer Assisted Learning, 17, 21-33. 

Chiu, C.H., Wu, W.S., & Huang, C.C. (2000). Collaborative concept mapping processes mediated by computer. 

Institute of Computer and Information Education, National Taiwan Teachers College, 33 (2), 95-100. 

Cifuentes, L., & Hsieh, Y. C. (2003a). Visualization for construction of meaning during study time: A Quantitative 

Analysis. International Journal of Instructional, Media, 30 (3), 263-273. 

Cifuentes, L., & Hsieh, Y. C. (2003b). Visualization for construction of meaning during study time: A Qualitative 

Analysis. International Journal of Instructional Media, 30 (4), 407-417. 

Cifuentes, L., & Hsieh, Y. C. (2004). Visualization for middle school student’ engagement in science learning. 

Journal of Computers in Mathematics and Science Teaching, 23 (2), 109-137. 

Daley, B. J. (2002). Facilitating learning with adult students through concept mapping. The Journal of Continuing 

Higher Education, 50 (1), 21-31. 

Duffy, T.M., Lowyck, J., & Jonassen, D.H. (1991). Designing environment for constructive learning, New York: 

Springer. 

Earnshaw, R.A., & Wiseman, N. (1992). An introductory guide to scientific visualization, New York: Springer. 

Emerson, R. M., Fretz, R. I., & Shaw, L. L. (1995). Writing ethnographic fieldnotes, Chicago, IL: The University of 

Chicago Press. 

Fischer, F., Bruhn, J., Grasel, C., & Mandl, H. (2002). Fostering collaborative knowledge construction with 

visualization tools. Learning and Instruction, 12, 213-232. 

Fischer, K.M. (1990). Semantic-networking: The new kid on the block. Journal of Research in Science Teaching, 27, 

1002-1018. 

Gaines, B.R., & Shaw, M.L.G. (1995). Concept maps as hypermedia components. International Journal of Human- 

Computer Studies, 43 (3), 323–361. 

Gobert, J.D., & Clement, J.J. (1999). Effect of student-generated diagrams versus student-generated summaries on 

conceptual understanding of causal and dynamic knowledge in plate tectonics. Journal of Research in Science 

Teaching, 36 (1), 39-53. 

Horton, P.B., MacConney, A.A., Gallo, M., Woods, A. L., Senn, G.J., & Hamelin, D. (1993). An investigation of the 

effectiveness of concept mapping as an instructional tool. Science Education, 77, 95-111. 

Hsieh, Y. C. & Cifuentes, L. (2003). A cross-cultural study of the effect of student-generated visualization on middle 

school students' science concept learning in Texas and Taiwan. Educational Technology Research and Development, 

51 (3), 90-95. 

Hsieh, Y.C., & Cifuentes, L. (2006). Student-generated visualization as a study strategy for science concept learning. 

Educational Technology and Society, 9 (3), 137-148. 

Hyerle, D. (2000). A field guide to using visual tools, Alexandria, VA: Association for Supervision and Curriculum 

Development. 

279

Jegede, O.J., Alaiyemola, F.F., & Okebukola, P.A.O. (1990). The effect of concept mapping on students' anxiety and 

achievement in biology. Journal of Research in Science Teaching, 27, 951-960. 

Jonassen, D. (1994). Thinking technology. Educational Technology, 34 (4), 34-37. 

Jonassen, D.H. (2000). Computer as mindtools for schools: Engaging critical thinking, Upper Saddle River, NJ: 

Prentice Hall. 

Jonassen, D. H., Carr, C., & Yueh, H.P. (1998). Computers as mindtools for engaging learners in critical thinking. 

TechTrends, 43 (2), 24-32. 

Lumpe, A. T., & Staver, J.R. (1995). Peer collaboration and concept development: Learning about photosynthesis. 

Journal of Research in Science Teaching, 32 (1), 71-98. 

Mayer, R. (1989). Systematic thinking fostered by illustration in scientific text. Journal of Educational Psychology, 

81, 240-246. 

Mayer, R., & Gallini, J. (1990). When is an illustration worth then thousand words? Journal of Educational 

Psychology, 82, 715-726. 

Merriam, S. B. (1998). Qualitative research and case study applications in education, San Francisco: Jossey-Bass. 

Novak, J. D. (1990). A useful tool for science education. Journal of Research in Science Teaching, 27 (10), 937-949. 

Novak, J.D. (1998). Learning, creating and using knowledge: Concept maps as facilitative tools in schools and 

corporations, Mahwah, NJ: Lawrence Erlbaum. 

Novak, J.D., & Gowin, D. B. (1984). Learning how to learn, Cambridge, UK: Cambridge University Press. 

Padilla, M. J., Miaoulis, I., & Cyr, M. (2002). Prentice Hall science explorer, Upper Saddle River, NJ: Prentice Hall. 

Plotnick, E. (1997). Concept mapping: A graphical system for understanding the relationship between concepts, 

retrieved October 15, 2007, from http://www.ericdigests.org/1998-1/concept.htm. 

Quinn, H. J., Mintzes, J. J., & Laws, R. A. (2003). Successive concept mapping. Journal of College Science 

Teaching, 33 (3), 12-16. 

Rice, D. C., Ryan, J. M., & Samson, S. M. (1998). Using concept maps to assess student learning in the science 

classroom: Must different methods compete? Journal of Research in Science Teaching, 35 (10), 1103-1127. 

Roth, W.M. (1994). Students’ views of collaborative concept mapping: An emancipatory research project. Science 

Education, 78 (1), 1-34. 

Royer, R., & Royer, J. (2004). Comparing hand drawn and computer generated concept mapping. Journal of 

Computers in Mathematics and Science Teaching, 23 (1), 67-81. 

Ruiz-Primo, M. A., & Shavelson, R. (1996). Problems and issues in the use of concept maps in science assessment. 

Journal of Research in Science Teaching, 33, 569-600. 

280

Hastie, M., Chen, N.-S., & Kuo, Y.-H. (2007). Instructional Design for Best Practice in the Synchronous Cyber Classroom. 


Instructional Design for Best Practice in the Synchronous Cyber Classroom 

Megan Hastie 

Brisbane School of Distance Education, Australia // Tel: +07 3214 8265 // mhast5@eq.edu.au 

Nian-Shing Chen and Yen-Hung Kuo 

Department of Information Management, National Sun Yat-sen University, Taiwan // nschen@cc.nsysu.edu.tw // 

d934020001@student.nsysu.edu.tw 

ABSTRACT 

This paper investigates the correlation between the quality of instructional design and learning outcomes for 

early childhood students in the online synchronous cyber classroom. Today’s generation of e-learners has 

access to highly engaging and well-designed multi-media synchronous classrooms. However little data exists 

on what constitutes ‘good practice’ in instructional design for online synchronous cyber lessons. The 

synchronous cyber classroom outperforms all other modes of instruction in enabling students to simultaneously 

integrate visual, auditory and kinaesthetic processes. The online synchronous cyber classroom provides learners 

with more authentic and engaging learning activities enabling higher levels of learning compared to purely 

asynchronous modes of self-paced learning. During 2001-2007 a group of students aged 5 to 8 years 

collaborated with their teacher at Brisbane School of Distance Education, Australia in a trial of online 

synchronous learning. The trial identified ‘best practice’ in the instructional design of synchronous lessons 

delivered through the Collaborative Cyber Community (3C) learning platform at the National Sun Yat-sen 

University, Taiwan. A guideline for ‘best practice’ in the instructional design of online synchronous cyber 

lessons for early childhood students has been developed and discussed. 

Keywords 

Instructional design, Online synchronous learning, Cyber classroom, Early childhood students 

Introduction 

This paper describes the evolution of ‘best practice’ in instructional design in synchronous cyber classrooms for 

students aged 5 to 8 years. The students were enrolled at Brisbane School of Distance Education (BSDE) and 

worked synchronously with their teacher Ms Megan Hastie during a six year trial. The trial became an international 

collaboration between BSDE, Australia and the National Sun Yat-sen University (NSYSU), Taiwan during 2005- 

2007. Synchronous teaching and learning over the Internet was embraced by the teacher and students as a means of 

overcoming the tyranny of distance and isolation experienced by most students. 

The paper attempts to define ‘best practice’ in instructional design for maximum learning gain in early childhood 

students in the synchronous cyber classroom. We examine instructional design within the context of the practical 

purpose of learning. We describe new instructional design elements that we have developed to maximize online 

learning for very young students. These elements acknowledge the prior learning of each student and provide the 

practical means for achieving expected learning outcomes. 

Technological innovation has provided educators with hardware and software but has not necessarily provided 

innovative instruction and pedagogy. To use an analogy, we have the machine but we are still waiting for the 

teaching manuals to be written. In particular, a paucity of data exists on instructional design for synchronous cyber 

learning with early childhood students. 

BSDE is a public school which operates under the governance of Education Queensland (EQ). Seven schools of 

distance education are located throughout Queensland. The students in this study were aged 5 to 8 years and were 

enrolled in Megan Hastie’s class at BSDE between 2001 and 2007. The students and their parents live in various 

locations throughout Australia and overseas. Students enrolled at BSDE access a range of asynchronous course 

materials including print, audio and multi-media. Students work off-campus, usually at home and under the 

supervision of a parent who is their Home Tutor. Students complete the course work and return the work to their 

teacher for evaluation. Communication between the student and the teacher has traditionally been by mail and 

telephone. With the advent of the Internet communication is occurring increasingly through email and web-based 






281

esources. The use of synchronous teaching and learning is being explored as a viable alternative and adjunct to 

traditional modes of delivery for students enrolled in Education Queensland schools of distance education. 

Nowadays, people are living in the global village such that children may go with their parents to a foreign country 

because of changes in their work locations. With the global Internet environment, children can use Information 

Communication Technologies (ICT) to keep in touch with their family and friends. Furthermore, e-learning can also 

help individual students continue their education. Interaction is one of the key factors in gaining access to 

information and for effective learning (Keegan, 1990). However, instructors are supposed to take more responsibility 

for instructional design to improve and encourage learners in online learning activities (Hannafin, 1992). 

Given that ICT enable the instantaneous transfer of information; educators have the added challenge of designing 

instruction for high-speed interaction. In ICT-enabled educational environments thoughts can be transferred from 

one person to another in nano-seconds enabling brain-to-brain information exchange (Hastie & Chen, 2006). The 

synchronous component of ICT-enabled learning environments is gaining more prominence. For instance, Chen et al 

(2005) state that the synchronous cyber classroom provides a learning environment that can outperform both 

asynchronous online instruction and traditional face-to-face instruction. Little attention, however, has been given to 

the use of synchronous cyber learning with younger students and to the development of ‘best practice’ in 

instructional design in the synchronous cyber classroom. This paper attempts to redress the situation. 

Literature Review 

Despite current behaviorist-oriented or constructivist-oriented arguments, instructional design makes instructors take 

a systematic approach to creating a learning environment that promotes effective learning outcomes. In this paper we 

define instructional design as the process through which an educator determines the best teaching methods for 

specific learners in a specific context, attempting to obtain a specific goal (Dick & Carey, 2001). 

The challenge for today’s educators is to develop ‘best practice’ in instructional design for lessons in the 

synchronous cyber classroom. Chickering, & Ehrmann (1996) say that simply incorporating technology into 

instruction is not enough. Instructors need to focus on seven ‘best practice’ strategies: 

1. Increase interaction between instructors and students. 

2. Increase cooperation among students. 

3. Increase students´ active learning. 

4. Prompt feedback given to students. 

5. Facilitate students´ time on task. 

6. Communicate expectations. 

7. Adapt to students with diverse talents and ways of learning. 

These strategies are highly relevant to the pedagogy of the synchronous cyber classroom. The challenge for 

educators is to ensure that ‘best practice’ is translated into ‘real’ practice in synchronous cyber lessons. 

Worley (2000) cited Ehrmann in arguing that the learner should be the focal point in research on web-based learning. 

In particular she says that the relationship between faculty and student deserves more investigation. Greater 

prominence should be given to the specific learning strategies rather than the impact of the technology in isolation. 

Witt & Wheeless (2001) say that research must consider other variables, particularly those related to instructional 

strategies to determine the effectiveness of web-based learning. 

A high correlation has been identified between the immediacy of verbal and nonverbal responses and cognitive 

learning (Rodriguez et al., 1996). The immediacy of the teacher’s verbal and nonverbal behaviours in face-to-face 

situations has been linked both directly and indirectly to enhanced cognitive and affective learning. Immediacy 

therefore is equally relevant, and possibly more relevant, in web-based lessons because participants are required to 

create a social presence through their communications. The synchronous cyber classroom lends itself to high speed 

verbal and nonverbal interaction between teacher and student. When translated into practice in the synchronous 

cyber classroom, the teacher and student use the technology to establish their communication link, to create a social 

presence and negotiate the lesson content. They then embark on the ‘real’ synchronous interactive component of the 

lesson using the system tools. The teacher gives the student ‘live’ feedback and evaluation. Recordings of the lesson 

282

can be used by both the teacher and student for evaluation of the learning process. Storage of recorded lessons means 

instructional design issues can be identified and resolved. 

With feedback gained from the instructional design process, refinements can be made to the adopted learning theory 

and to the instructional process itself. In computer-mediated instructional design (including e-learning, e-tutoring, 

course development.), instructional design plays a critical role in enabling immediacy of response and in creating 

social presence. The immediacy behaviors of participants in the synchronous cyber classroom become more 

prominent. The role of the teacher shifts from discussion leader to discussion facilitator as the student assumes more 

responsibility. This facilitates technology-based learning which enables students to solve their specific learning 

problems (Nichols & Anderson, 2005). 

Cognitive gain is one of the important goals for instructional design (Chial, 2004; Philips, 1994). Teachers, therefore, 

need to focus on how to design learning activities which can better engage students in active learning. This results in 

deeper learning and promotes cognitive gain Topping (2005) has pointed out that cooperative learning and peer 

tutoring facilitated by ICT tools can increase students’ self-organizing opportunities and improve cognitive gain 

(Topping, 2005; Topping et al., 2004). Other literature has also reported on ways to use different ICT tools to 

promote self-organizing opportunities for learners (Vogel, et al., 2006; Corpus & Eisbach, 2005). 

Synchronous learning methodology has been acknowledged as an important strategy for collaborative learning 

(Marjanovic, 1999). Chen, Ko, Kinshuk, & Lin (2005) demonstrate that the most promising advantages of the 

synchronous cyber classroom are in the provision of immediate feedback such that learners in the cyber classroom 

are able to correct themselves immediately and thereby strengthen their learning. The synchronous process enhances 

student motivation through the students’ obligation to be present and to participate in the face-to-face cyber 

environment. Wang & Chen (2007) describe the results of a pilot study assessing the value of a synchronous learning 

management system to support online live tutorial sessions in second language learning. They found that the range of 

tools supported by the platform including chat, whiteboard, and videoconferencing technology can provide a 

resilient, supportive learning environment for distance learning students. Hastie & Palmer (2003) found that the 

synchronous cyber classroom demands higher levels of concentration in learners during learning activities and 

thereby maximizes memorization and learning outcomes. The results recorded in the synchronous cyber classroom 

are compared to its asynchronous cyber classroom counterpart. The paper concludes that there is a need for educators 

to give greater prominence to synchronous cyber lessons for learners aged 5-8 years instead of using the current 

pedagogy of starting from asynchronous learning activities. 

Methodology 

The trial provided individualised instruction to students aged 5-8 years using online synchronous cyber classrooms 

created by the digital school called the Collaborative Cyber Community (3C). The digital school server is located at 

NSYSU and is a kind of Synchronous Learning Management System (SLMS) developed and supported by Professor 

Nian-Shing Chen to facilitate international team-teaching for young children. The teacher, Megan Hastie, was based 

in Brisbane. The students were located within Australia and throughout the world, including Asia and the Pacific, 

Europe and the United States of America. 

The synchronous cyber lessons were an adjunct to the regular program provided at BSDE. The students volunteered 

to participate in the trial. Students worked at home using a personal computer and linked-up to the Collaborative 

Cyber Community digital school platform. 

The early phase: communication with isolated learners 

In the early phase of the trial the synchronous cyber classrooms were used to enable live communication between the 

teacher and students. The cost of telephone calls to isolated and international students meant that these students 

seldom had the opportunity to speak to their teacher. The majority of their communication was asynchronous and 

paper based. Communication relied on traditional mail services. 

283

The use of digital communication modes via the Internet became a feature at BSDE in 2000. This enabled email 

contact and more significantly provided broadband capacity that supported an interactive whiteboard and Voice over 

Internet Protocol (VoIP). This enabled the teacher and students to talk and interact with each other in ‘real time’. 

The use of the Internet for communication set a precedent in itself because many of the students were geographically, 

socially and educationally isolated. This meant that the students went from practically no direct contact and 

communication with their teacher to ‘real’ time interaction and discussion. 

The teacher and student entered the synchronous cyber classroom by logging-on to the Collaborative Cyber 

Community platform. The platform supports both asynchronous and synchronous functionalities for teachers to 

design and conduct various teaching & learning activities in asynchronous cyber classroom or synchronous cyber 

classroom. The student and their home tutor, usually a parent, logged-on to the Collaborative Cyber Community 

platform at a pre-arranged time. Allowances were made for time zone differences for students living overseas. 

Both teacher and student wore a headset with built-in microphone. The headsets limited extraneous noise and audio 

feedback. The interactive whiteboard was used, along with the webcam (where available) for the sharing of digital 

information. VoIP enabled audio and discussion. The student used a mouse or a graphic tablet to draw and write on 

the synchronous interactive whiteboard. The teacher provided the student with digital and auditory feedback in the 

form of written and verbal comments. A webcam (where available) was used by the teacher and student. The 

teacher was able to make direct live observations of the student at work. 

The interaction between the teacher and student in the initial phase of the trial can best be described as teacherdirected 

learning. The teacher prepared multiple whiteboard screens prior to the lesson. The screens featured 

activities designed to meet the specific learning needs of the individual learner. This type of activity resembled a 

‘worksheet’ similar to the paper activities found in traditional classrooms. The teacher used commercially produced 

graphics as picture clues. The student completed the activity and received immediate feedback from the teacher in 

the form of a ‘tick’ and written praise for a correct answer. Negative feedback was avoided for incorrect answers. 

Rather the student was encouraged to have another attempt at the correct answer. This allowed the student to perform 

tightly scripted tasks in a teacher-directed and highly supportive learning environment. An example of this type of 

activity is shown in Figure 1. 

Figure 1. Worksheet format 

The Second Phase: student and teacher ‘real’ collaborative learning 

The communication capabilities of the synchronous cyber classrooms were established in the early phase of the trial. 

The focus of the trial then changed to the pedagogical issues related to synchronous teaching and learning. This can 

be largely attributed to a student named Madeline. 

284

Madeline was eight years of age and worked online in a synchronous cyber classroom with Megan Hastie during 

2001-2002. A gifted student, Madeline brought a new dimension to cyber lessons. Her written responses on the 

interactive whiteboard in the synchronous cyber classroom provided irrefutable evidence of abstract thinking and 

higher levels of cognitive functioning. Madeline created ‘mind maps’ on the interactive whiteboard to record and 

plan her learning. This is a form of ‘metacognition’ in which Madeline demonstrated her ability to think about her 

thinking. She recorded her thoughts on the interactive whiteboard whilst simultaneously typing the text and 

discussing the content with her teacher. Figure 2 shows Madeline was planning a research project on birds that were 

nesting on her parents’ farm: 

Figure 2. High level cognitive function 

Figure 3. High level cognitive tasks 

From this point the trial sought to maximise the interaction between the student and teacher during synchronous 

cyber lessons. A less didactic approach was adopted. Strategies were trialled to support instruction that demanded 

demonstrated evidence of visual, auditory and kinaesthetic processing by the student during the lesson. We expected 

the integration of these three sensory functions to result in higher levels of thinking and learning. This posed 

instructional design challenges. During 2002-2007 the design of the whiteboard screens evolved such that the student 

285

contributed more information and engaged in ‘chat’, both verbal and written, relating to the activity. By contrast, the 

Figure 3 below shows how instructional design enabled the student to interact in a more spontaneous way. The 

student’s responses were visual, auditory and kinaesthetic. This demanded a higher level of thinking and contributed 

to cognitive gain. 

We found a direct correlation between the instructional design features illustrated in Figure 3 and higher levels of 

interactivity by the student. The student provided more information and demonstrated greater levels of understanding 

of the concepts. 

We attribute the teacher’s ability to adopt a less didactic and teacher-directed approach to the design of synchronous 

cyber lessons to the teacher gaining confidence with the technology of the synchronous cyber classroom. This 

included greater competence in the establishment of the link with the student, the use of the interactive whiteboard 

and its tools, and the design of synchronous cyber lessons. Growing confidence with the technology allowed the 

focus to shift to the pedagogy. 

This resulted in higher levels of interaction and collaboration between the student and teacher during the lesson. It 

meant the teacher and student shared the learning space on the interactive whiteboard. The whiteboard became a 

colourful, creative and dynamic playground for learning. Input from the student was maximised and resulted in less 

teacher preparation time. The teaching and learning became ‘real’. The evolution of the lesson format and its 

associated instructional design features was guided by the students themselves. 

We decided to keep the instructional design for the synchronous lessons simple and ‘minimalist’. The focus was on 

clarity of communication between the student and teacher and was deemed to be the most suitable approach for 

working with younger students. It provided the students with the freedom to use concrete technological tools, 

especially the writing tools on the interactive whiteboard, to encode abstract thought. An interesting dichotomy 

evolved between simplicity in the instructional design and complexity in the cognitive functioning of students. In 

effect, the teacher and students collaborated in the learning process and the design of lessons. This approach had two 

major advantages: it was a good way to engage online students, and it was also a good strategy to encourage 

teachers to take a graduated approach to the adoption of online synchronous teaching. 

The simplicity of the approach with its ‘minimalist’ design features was applied to the lesson format and also to each 

individual screen of the synchronous interactive whiteboard. The increased interaction by the student in the form of 

visual, auditory and kinaesthetic responses during synchronous cyber lessons was equated with higher cognitive 

function. 

The lesson format usually consisted of three or four screens prepared by the teacher prior to the lesson. Extra screens 

were added to the lesson as required. Some screens contained a simple one-sentence instruction or idea for the 

student to read. The remaining space on the screen was purposely left empty and was used for drawing and writing 

by the student and teacher. Other screens were completely empty and were added as the lesson progressed to provide 

extra writing space for the student. 

As a result, we developed a ten point guideline for ‘best practice’ in instructional design to maximise learning 

outcomes in the students in our trial: 

1. We kept the design simple (minimalist) when we planned the format of the synchronous lesson and for each 

screen of the interactive whiteboard. 

2. We used the first screen of the synchronous whiteboard, along with the webcam (where available) to start the 

lesson and welcome the student. 

3. We used the second screen for the lesson plan. 

4. We provided teacher directed activities during the lesson. 

5. We used the third and subsequent screens as a working space for activities based on the lesson plan. 

6. We gave greater prominence to freehand drawing and keyboard writing on the interactive whiteboard. 

7. We balanced interactivity and spontaneity with teacher ‘wait time’. 

8. We used the final screen to plan the next lesson. 

9. We also used the final screen to praise the student, end the lesson and say farewell. 

10. We used email as an adjunct to synchronous cyber lessons. 

286

We applied the ten points to all synchronous cyber lessons with points 1, 2, 3, 8, 9, 10 forming a standard template. 

This provided students with a predictable format for the lessons and minimised teacher preparation time. The 

students formed an expectation that the lesson would start with a greeting and a discussion of the lesson plan (the 

advance organiser). Points 4-7 formed the working space for collaborative learning between the student and teacher. 

From this stage of the lesson until its conclusion the expectation of the students was that they would work 

collaboratively with their teacher on self-selected and teacher directed tasks. The teacher was able to provide direct 

instruction while facilitating the shared ownership of the working space with the students. The lesson conclusion 

correlates with points 8-10 with acknowledgement of students’ effort and learning outcomes and collaborative 

planning for the next lesson. We describe the guideline in more detail in the following section. 

Findings & Discussions 

We have identified ten basic instructional design elements for synchronous cyber lessons with students aged 5-8 

years. These will now be described in detail: 

1. Keep the design simple (minimalist) for the lesson format and for each screen of the interactive cyber 

whiteboard 

• Use the student’s name. 

• Encourage the student to respond using your name. 

• Give thought to your ‘persona’ as a synchronous cyber teacher and try to simply be yourself. 

• Keep verbal and written communication simple for clarity of communication. 

2. Start the lesson and welcome the student 

• Use the first screen of the interactive whiteboard to greet the student and start the lesson. 

• Use webcam, if available, to greet the student and their Home Tutor. 

• Use the student’s name in both the written and verbal greeting. 

• Continue to use the student’s name throughout. 

• Use one colour for the greeting (blue is good). Write the greeting at the top of the screen. 

• Continue to use the same colour throughout the lesson. 

Figure 4. An example of element 2 

287

3. Use the second screen for the lesson plan 

• Provide the student with an advance organizer based on negotiated content. 

• Invite the student and their Home Tutor to nominate content and specific learning for the lesson. 

• Confirm the lesson plan with the student and Home Tutor at the start of the lesson or prior to the lesson via 

email 

Figure5. An example of element 3 


4. Provide teacher directed activities during the lesson 

• Provide pedagogical input for decisions about content based on observations of the student’s progress, child 

development theory and curriculum requirements. 

• Use simple written instructions throughout the lesson, preferably one sentence, placed at the top of the screen. 

288

• Use graphics and photographs as picture clues. 

• Share ownership of the empty space on the whiteboard with the student. 

• Draw a box on the whiteboard screen to define the working space for more structured activities. 

• Use a different colour for feedback and comments to the student (pink is good). 

• Write feedback and comments at the bottom or side of the screen or wherever a space can be found that does not 

overlap the student’s written work. 

5. Use the third and subsequent screens as a working space for activities based on the lesson plan 

• Use the whiteboard as a learning space and ‘playground’. 

• Add screens as required. 



6. Give greater prominence to freehand drawing and keyboard writing on the whiteboard 

• Encourage the student to draw pictures on the whiteboard as a way of developing confidence. 

289

• Use drawing to develop writing skills. 

• Encourage the student to use the tools in the toolbar. 

• Encourage the student to experiment with colour using the palette. 

• Use writing to develop reading skills. 

• Encourage the student to develop keyboarding skills. 

• Encourage the student to use the graphic tablet. 

7. Balance interactivity and spontaneity with teacher ‘wait times’. 

• Allow the student time to respond verbally. 

• Keep the teacher’s verbal comments simple to accommodate audio lag. 

• Write comments and feedback on the whiteboard to compensate for audio lag. 

• Repeat verbal comments in writing to restate the message if necessary. 

• Use the chat room to communicate with the Home Tutor. 


Figure 10. An example of element 8 

290

8. Use the final screen to plan the next lesson 

• Negotiate the content for the next lesson with the student and their Home Tutor. 

• Provide pedagogical input for decisions about content based on observations of the student’s progress, child 

development theory and curriculum requirements. 

9. Use the final screen to praise the student, end the lesson and say farewell 

• Give positive feedback to the student. For example, ‘You did clever writing today.’ 

• Tell the student that the lesson is ending 

• Say farewell. For example, ‘Bye for now. I’ll talk to you again soon’. 


10. Use email as an adjunct to synchronous cyber lessons 

• Use email to confirm the lesson 

• Use email to give feedback to the student and Home Tutor after the lesson. 

Throughout our trial of synchronous cyber teaching and learning with students aged 5-8 years we found an increased 

level of interaction between teacher and students. We negotiated the lesson content with the student and their Home 

Tutors. We used an advance organiser at the start of each lesson to communicate expectations. We were able to 

cater for students with diverse talents and ways of learning through negotiated curriculum and ‘tailor-made’ lessons 

to suit individual needs. We found that the synchronous interactive whiteboard compelled the student and teacher to 

encode information in a high speed exchange of written responses. Students used the mouse, graphic tablet and 

keyboard to write on the synchronous interactive whiteboard. Students acquired sophisticated keyboarding skills 

from an early age, generally by age seven years. Keyboarding enabled the students to participate in the learning 

process during synchronous cyber lessons in a highly interactive manner. 

We believe that the level of interactivity demonstrated by the students in our trial during synchronous cyber lessons 

was superior to any other learning environment, including face-to-face. The teacher was able to respond 

immediately to the student using the mouse and keyboard to provide written instructions and feedback to the student 

on the synchronous interactive whiteboard. The multi-layered and colourful written interactions created by the 

291

student and teacher on the synchronous interactive whiteboard became the ‘face’ of the communication and 

contributed to the evolution of a social presence that we believe is unique to synchronous cyber lessons. We found 

that the immediacy of the teacher’s verbal and nonverbal behaviours during intense, high speed interaction between 

the student and teacher resulted in accelerated learning by the students. The tools of the synchronous cyber 

classroom were observed to contribute to a higher level of efficiency in the mastery of concepts and the completion 

of tasks by the students. In particular, the students demonstrated higher levels of concentration and increased work 

rates. We found that the quality and quantity of the work completed by the students, for example, in twenty minute 

online synchronous lessons could be compared to longer time allocations in traditional classroom settings. The 

students’ time on task was maximised and resulted in enhanced cognitive and affective learning. In terms of 

engaging students, maintaining high levels of concentration, capitalising on their individual interests and learning 

styles and simply ‘getting-the-job-done’ we found synchronous lessons surpassed all other modes of instruction. 

Although we worked individually with our students we found that the students, who were geographically isolated 

from their teacher and from one another, were beginning to form a learning community within the Youth Knowledge 

Network. We anticipate that the students’ technologically networked community will continue to grow and that this 

will result in increased cooperation among our students in the future. 

Our guideline for ‘best practice’ in instructional design in synchronous cyber classrooms is a practical application of 

the seven strategies identified by Chickering, & Ehrmann (1996). Chickering & Ehrmann suggested educators use 

technology in instruction to increase interaction between instructors and students, to increase cooperation among 

students, to ,increase students´ active learning, to provide prompt feedback to students, to facilitate students´ time on 

task, to communicate expectations and to provide benefits for students with diverse talents and ways of learning. The 

synchronous cyber classroom provided the technological tools to apply the strategies as described by Chickering and 

Ehrmann (1996). But we found there was no manual for teachers to use during synchronous cyber lessons that gave 

a practical and pedagogically sound guide to the instructional design of such lessons. The guideline was developed, 

therefore, as a ‘user friendly’ checklist for teachers who are embarking on synchronous cyber teaching. 

In summary, our major finding was the enhanced learning outcomes that we observed in all the students in our trial. 

We attribute this to the ideal learning environment provided in the synchronous cyber classroom. Students developed 

higher levels of concentration and memorization as a direct result of the uniquely individual learning process that 

occurs in synchronous cyber lessons. Students were able to integrate visual, auditory and kinesthetic processes free 

from the distractions that plague the traditional classroom. We were able to record and quantify the students’ 

learning outcomes as evidenced in their written and verbal responses. Based on our findings we believe an urgent 

need exists for further research to be undertaken into brain function in early childhood students during lessons in the 

synchronous cyber classroom. This leads us to the conclusion that synchronous cyber teaching and learning that is 

informed by best practice in instructional design poses a serious challenge to traditional classroom practices in early 

childhood education. 

Conclusion 

A paucity of literature exists on best practice in instructional design for online synchronous cyber lessons with 

students aged 5-8 years. This paper has attempted to identify ‘best practice’ in instructional design in the online 

synchronous cyber classroom. We equated increased interactivity by students in the form of verbal and written 

responses during synchronous cyber lessons with higher learning outcomes. We developed a guideline for the 

instructional design of synchronous cyber lessons that would maximise student interaction and result in enhanced 

learning. The guideline is, therefore, a practical application of best practice strategies and a survival manual for 

teachers embarking on synchronous cyber teaching. 

Essentially we found that when the teacher adopted a simplified and ‘minimalist’ approach to instructional design: 

the students contributed significantly more information and demonstrated higher levels of learning. We regard this as 

‘real’ collaborative learning. The students’ rate of response was faster and involved an integration of visual, auditory 

and kinaesthetic processes. We attribute this to the unique and ideal learning environment that is created in the 

synchronous cyber classroom. 

We believe the simplified ‘minimalist’ approach can be used to encourage more teachers to embark on synchronous 

online teaching as it helps build confidence in what may be perceived as a highly innovative yet challenging 

292

application of technology. It is a carefully considered pedagogical approach to working synchronously with early 

childhood learners. As such it has the potential to influence best practice in synchronous cyber teaching and learning. 

In conclusion we say the synchronous cyber classroom outperforms all other modes of instruction. We urge 

educators to give greater prominence to the synchronous component of online teaching and learning. 


This study was supported by the National Science Council, Taiwan, under grant NSC95-2520-S-110-001-MY2. 

References 

Chen, N.S., Ko, H. C., Kinshuk, & Lin, T. (2005). A model for synchronous learning using the internet. Innovations 

in Education and Teaching International, 42 (2), 181-194. 

Corpus, J. H. & Eisbach, A. OD.. (2005). A live demonstration to enhance interest and understanding in child 

development. Journal of Instructional Psychology, 32 (1), 35-43. 

Chial, M. R. (2004). A brief guide to instructional development, retrieved October 15, 2007, from 

http://www.comdis.wisc.edu/staff/mrchial/InstDevSite/index.htm. 

Chickering, A. W., & Ehrmann, S. C. (1996). Implementing the seven principles: technology as lever. AAHE 

Bulletin, 49 (2), 3-6. 

Dick, W., & Carey, L., & Carey, J. O. (2001). The systematic design of instruction, NY: Addison-Wesley. 

Hannifin, M. J. (1992). Emerging technologies, ISD and learning environments: critical perspectives. Educational 

Technology Research & Development, 40 (1), 49-63. 

Hastie, M., & Chen, N.S. (2006). Working brain-to-brain: ‘real learning’ - teacher-directed online live lessons using 

a synchronous cyber classroom. Paper presented at the Australian Computers in Education Conference 2006 (ACEC 

2006), October 2-4, 2006, Cairns, Australia. 

Hastie, M., & Palmer, A. (2003). ‘Real time, real young, real smart’- the use of the internet for real time teaching 

with 5 to 8 year olds. Paper presented at the Open Distance Learning Association of Australia Conference (ODLAA 

2003), October 1-4, 2003, Canberra, Australia. 

Keegan, D. (1990). The foundations of distance education, London: Routledge. 

Marjanovic, O. (1999). Learning and teaching in a synchronous collaborative environment. Journal of Computer 

Assisted Learning, 15 (2), 129-138. 

Nichols, M., & Anderson, B. (2005). Strategic e-learning implementation. Discussion paper of the International 

Forum of Educational Technology & Society, retrieved October 15, 2007, from http://ifets.ieee.org/discussions/ 

discuss_july2005.html. 

Phillips, L. (1994). The continuing education guide –the CEU and other professional development criteria, Dubuque, 

IA: Kendall/Hunt. 

Rodriguez, J. L., Plax, T. G., & Kearney, P. (1996). Clarifying the relationship between teacher nonverbal 

immediacy and student cognitive learning: affective learning as the central causal mediator. Communication 

Education, 45, 293-305. 

Topping, K. J., Peter, C., Stephen, P., & Whale, M. (2004). Cross-age peer tutoring of science in the primary school: 

influence on scientific language and thinking. Educational Psychology, 24 (1), 57-76. 

293

Vogel, J. J., Vogel, D. S., Cannon-Bowers, J., Bowers, C. A., Muse, K., & Wright, M. (2006). Computer gaming and 

interactive simulations for learning: a meta-analysis. Journal of Educational Computing Research, 34 (3), 229-243. 

Wang, Y., & Chen, N.S. (2007). Online synchronous language learning: SLMS over the Internet. Innovate, 3 (3), 

retrieved October 15, 2007, from http://innovateonline.info/index.php?view=article&id=337. 

Witt, P. L., & Wheeless, L. R. (2001). An experimental study of teacher’s verbal and nonverbal immediacy and 

student’s affective and cognitive learning. Communication Education, 50 (4), 327-342. 

Worley, R. B. ( 2000). The medium is not the message. Business Communication Quarterly, 63, 93-10. 

294

Malinski, R. (2007). Book review: Cases on global e-learning practices: successes and pitfalls (Eds. R.C. Sharma and S. Mishra). 


Reviewer: 

Richard Malinski 

Ryerson University, Canada 

richard@ryerson.ca 


(Book Review) 

Textbook Details: 


2007, R.C. Sharma and S. Mishra (Eds.) 

Information Science Publishing, Hershey, PA 17033, USA 

ISBN - 1-59904-340-8, 356 pages 

The table of contents is available on the website 

http://www.igi-pub.com/books/additional.asp?id=6164&title=Table+Of+Contents&col=contents 

The preface is available online at 

http://www.igi-pub.com/books/additional.asp?id=6164&title=Preface&col=preface 

The first chapter can be downloaded from 

http://www.igi-pub.com/books/additional.asp?id=6164&title=Book+Excerpt&col=book_excerpt 

Introduction 

The authors state that the object of their book is ‘to provide learning opportunities through a set of case studies on 

implementation of e-learning’ (p vii). The editors accomplish this by providing 23 case studies divided into three 

sections, i.e. completely online learning systems (10 cases), blended online learning systems (8), and resources based 

online learning systems (5). To surround these cases the editors provide a brief introduction to e-learning and an 

even briefer summary of findings as conclusion. 

The introductory essay by the editors points out the lack of sound pedagogic practices in e-learning courses today 

and therefore the need for clarity and appropriate strategies. The editors miss a step towards clarity by not detailing 

what the significance is of having the three sections. They do however include a list of the many definitions of elearning 

and its synonyms and then define e-learning to be ‘teaching and learning in a networked environment with 

or without blending of face-to-face contact and other digital media’ (p 4). Once again the editors miss an 

opportunity for clarity by not going into detail and example to help specify what they really mean and how this fits 

with the categories of case studies. 

They do continue by listing some of the many benefits from e-learning and numerous factors one should consider in 

the design and development of e-learning. Most significant is the restatement of the ERIC (Experience-Reflect- 

Interact-Construct) framework outlined in an earlier work by Mishra (Mishra, 2002) and which integrates elements 

of behaviorism (content), cognitivism (learner support), and constructivism (learning activities). It may not be a 

novel framework but it is useful in organizing three significant elements of e-learning. Most of the case study 

authors mention these key elements but tend to focus on the importance of the constructivism and the provision of 

knowledge-building learning environments for students. 

The editors attempted to have the case study authors cover specific topics so that there would be some 

standardization of content and ease of comparison or assessment of case study information. They outlined their case 

study framework which included topics such as academic and administrative issues; program evaluation; networking 

and collaboration; policy implications; sustainability and conclusion along with lessons learned and best practices. 

The editors allowed some leeway in both the coverage of these topics as well as in writing styles and so readers need 

to be patient while reading some of the case studies. 






295

Cases 

The first set of ten case studies cover examples of completely online learning systems. Within these cases there 

tends to be a restatement of the benefits of e-learning along with the acceptance of the authentic learning and 

knowledge construction possibilities in e-learning. Most but not all cover the case study elements outlined by the 

editors, nevertheless there is much for those interested in or involved with e-learning. Several issues are of particular 

note in this first set of cases, e.g. 

Assessment - 

• Assessment tools not only for the e-learning courses but also for the instructors are very important. 

• The positive student assessment of the courses tends to be directly proportional to the participation of the 

instructor. 

Networking - 

• For those interested in program development networks and their issues, the German WINFOline discussion can 

serve as a sound model and an outline of some of the hurdles that crop when working with other institutions. 

Professional development - 

• For those just moving to e-learning, the New Mexico State University professional development program is a 

realistic and useful case. The lessons learned here are exemplary, e.g. the learning curve is steep, capitalize on 

established models, and be prepared for ‘Plan B.’ This is complemented by the later case on educational reform 

which outlines some very useful lessons and practices, such as preparing teachers with problem solving and 

technology awareness sessions as well as developing communications support structures to support feedback 

and discussions. 

• The New Jersey-Namibia teacher training collaboration case brings out cross-cultural and cross-time period 

issues. Specifically, there is a need for flexibility when networks are unstable or less advantaged students must 

rely on Internet Cafés for access. 

Student support - 

• The pilot program to introduce student e-portfolios illustrates the use of this learning tool in promoting 

reflection and self-understanding in a concrete practical way. 

• The Bridging Online (BOL) Program in the Co-Op unit at Simon Fraser University in British Columbia 

provides excellent examples of products helping co-op students become more confident and self-directed in 

understanding the pertinence and value of their skills for employment. 

The second set of eight cases covers those learning systems that the editors consider Blended Online Learning 

Systems. The editors do not make it clear how these cases differ from the first set as a result the distinction is more 

a curiosity than a help. This section brings out the difficulty of trying to pigeon-hole these e-learning case studies 

into distinct groups. The readers should not expect to see clearly discontinuous categories in this book because 

some of the cases could be put in one or more sections. Such cases are really examples along a continuum from less 

to more use of online resources and activities. Important here are the stories that the authors tell, the lessons that 

they learn, and the best practices that they suggest! Significant issues brought up in these eight cases are; 

Planning - 

• Timing always seems to be too short so do leave sufficient time to complete tasks 

• Funding is also something that requires much thought – e-learning is expensive so it is important to realize the 

limits of the e-learning program and make everyone aware of these. 

• Once again, be ready for surprises and unexpected results – remain flexible. 

• Understanding institutional policies and being ready to explain the e-learning program is essential. 

Student engagement - 

• Provide students with learning environments that bring together both theoretical and real-world scenarios. 

• Seek out assessments of e-learning directly from the students in order to improve the e-learning offerings. 

• Students are themselves a resource for learning. Use them to assist their fellow students. 

Instructor support - 

• There are many fundamental challenges facing instructors – changing roles with online activity especially the 

need for online support and participation, increased technical requirements, combining face-to-face with online 

contacts, perhaps increasing student load or development requirements. 

296

• Assistance in designing and working with authentic learning environments or ‘deep learning’ can provide 

instructors with insights into the potential of technology and into their pedagogy. 

The third set of five cases focus on Resource-Based Online Learning Systems. The editors do not define exactly 

what they mean by ‘Resource-Based Online Learning Systems’ so do not be misled by the section title. While these 

five cases note many of the same examples of lessons learned and best practices noted in the other cases, there are 

several aspects that are noteworthy in these cases. These particular aspects are; 

Costing - 

• The multimedia instructional product for medical school students mentions the need for experts in 

storyboarding and scripting as well as for technically skilled people to transform the content into online 

materials. The planning and development process outlined would be extremely useful. This case is the only 

one to mention costs - $75,000 per hour of multimedia course. 

Planning - 

• The Hard Fun case illustrates a hardcore constructivist approach using much jargon in developing a learning 

resource. The components that go into this resource and the evaluation rubric outlined are excellent examples 

and will be of use to those wanting to explore constructivism. 

• The ESPORT case brings out the importance of systematic and consultative evaluation of pilot projects and the 

significance of understanding the difficulties of adoption of innovations within organizations. 

• The EBS E-learning chapter reinforces the importance of orchestrating many factors in order for e-learning to 

be effective. 

• The last chapter discusses a multifaceted ideology which describes an e-learning ecosystem encompassing such 

elements as the conceptualization of courseware, standardizing interoperable content, and personalizing 

learning experiences. The two projects, one on student support and the other on faculty development, illustrate 

how useful such a framework is to integrating products into a unified program. 

The 23 cases give the reader different stories of how researchers around the globe are facing the challenges of elearning. 

While the stories are different many of the challenges faced are similar, i.e. planning ahead is critical, 

instructor development crucial, and student support vital. 

Conclusion 

This book is a valuable resource for practitioners but at the same time a difficult read! Editing a multi-chapter 

publication is a daunting venture and the editors should be commended for bringing these cases together. The 

editors did try to bring some standardization to the content formats but the variety of writing styles requires close 

reading to recognize what some of the authors really mean. Here is where the editors could have done more work 

in clarifying meaning, catching typographic errors, questioning vague terminology or unsupported conclusions, and 

improving illegible graphics. 

Nevertheless, the suggestions by the case authors are valuable for the lessons learned and suggestions for best 

practices. Readers interested in what others around the world are doing should find the cases useful for 

confirmation as well as for new insights. In addition, the experiences from around the world show that while some 

are ahead of others, instructors and students are facing the same challenges in e-learning. 

Reference 

Mishra, S. (2002). A design framework for online learning environments. British Journal of Educational 

Technology, 33 (4), 493-496. 

297

Kılıçkaya, F. (2007). Website review: WordChamp: Learn Language Faster. Educational Technology & Society, 10 (4), 298-299. 

Reviewer: 

Ferit Kılıçkaya 

Middle East Technical University, Faculty of Education 

Department of Foreign Language Education 

06531 Ankara, Turkey 

kilickay@metu.edu.tr 

http://www.metu.edu.tr/~kilickay 

http://www.technologycallsyou.com 

Site URL: 

http://www.wordchamp.com 

Site title: 



(Website Review) 

Objective of the site: 

The very basic objective is to help the teachers, students and organizations in the process of learning new vocabulary 

of the target language that they are aiming to learn. 

Intended audience: 

The site seems very useful to teachers and students of any language and the organizations aiming to help their 

employees and client. 

Domain related aspects: 

It is an educational site and it focuses on teaching vocabulary which is compatible with all languages. The site 

provides the audience with vocabulary of different types of drills including translation, listening comprehension, 

dictation, and language-specific drills. 

The content of the site is original and memberships including access to all features and content are free for everyone. 

Structure of the Site: 

“WordChamp: Learn Language Faster” is basically divided into three sections: 

Web reader 

Learn vocabulary 

Course management 

Browse languages 

Under the section headed “Web reader”, there is a page which help the audience read authentic texts. It takes any 

webpage or text and shows popup definitions to all the words it recognizes. It demolishes the frequent annoying 

time-consuming trips to the dictionary. Moreover, if the word has audio, the audience has the opportunity to hear a 

native speaker pronouncing it. “Learn vocabulary” hosts vocabulary drills including translation, listening 

comprehension, dictation, and language-specific drills. Also, the audience can create their own vocabulary lists with 

audio, which can be downloaded as mp3 files and as flashcards. You can find samples of the different drills here. 

The database currently holds 2,494,798 flashcards in 112 languages. “Course management” provides teachers with 

tools to help their students to learn vocabulary outside of class. With the tools provided in this section, teachers can 

create custom lists specific to their classes, set up a class and homework assignments. Moreover, they can also track 






298

their students’ performance. In “Browse Languages” section, the audience can go over the lists that have been 

prepared by others or browse the languages available. 

Usefulness and richness of each topic: 

Learning new vocabulary is one of the things that take the most time and the most of the learners have difficulty in. 

Vocabulary is inescapable since anything related to any language starts with learning new vocabulary. This site 

provides learners of any language with the vocabulary items (audio, definitions, drills, and flashcards). It makes it 

easy for students to practice the vocabulary they need. It is rich as regards the materials provided. 

Connectivity: 

I did not notice any problem in having access to the site through ADSL and standard modem connection. However, 

downloading the lists of vocabulary with audio is a problem with slow connection. No special software is needed. 

However, in order to listen to the audio files on this site and the lists of vocabulary items with audio, it is required to 

have a flash player and an mp3 player, which is freely available on the net. 

Interface related aspects: 

The layout of the website 

The layout is good and the links to the pages are clearly identifiable. There is no animations which are distracting. 

Site structure 

The audience can easily find what they are looking for via clear titles and links. There are separate sections for 

different aims (for students, teachers and organizations). The fonts are readable and the background color is not 

distracting. 

Navigation 

The audience can easily navigate the website. 

Search facilities 

The search facility provided helps the audience to search in the vocabulary lists, users and the words. 

Overall issues: 

The site is supported by a commercial firm and all the contact details of the site owners are provided. It has no 

distracting banners, animations or advertisements. 

The site is regularly updated and the news is provided through RSS. The external links to other language resources 

were valid at the time of writing this review. 

Other comments: 

“WordChamp: Learn Language Faster” is rich regarding the content, materials, vocabulary lists and the facilities 

provided to the learners and teachers of any language. The access to over 127,000 recordings of native speakers, the 

database currently holding 2,494,798 flashcards in 112 languages and “Web Reader” is really fascinating. 

Downloading lists and flashcards to mp3 players and to mobile phones, pre-made conjugation charts, and audio files 

make a huge difference while studying a foreign language. The tools provided by this site are invaluable. Better still, 

this site is currently open to access without any payment. However, audio is currently available in Arabic, Bulgarian, 

Chinese (Mandarin), English, Farsi, French, German, Hungarian, Italian, Japanese, Norwegian, Portuguese, Spanish, 

Swahili, Turkish, and Tagalog. 

299

October 2007 Volume 10 Number 4 - Educational Technology ...

Create successful ePaper yourself

Delete template?

Save as template?