29.03.2013 Views

October 2006 Volume 9 Number 4

October 2006 Volume 9 Number 4

October 2006 Volume 9 Number 4

SHOW MORE
SHOW LESS

You also want an ePaper? Increase the reach of your titles

YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.

<strong>October</strong> <strong>2006</strong><br />

<strong>Volume</strong> 9 <strong>Number</strong> 4


Educational Technology & Society<br />

An International Journal<br />

Aims and Scope<br />

Educational Technology & Society is a quarterly journal published in January, April, July and <strong>October</strong>. Educational Technology &<br />

Society seeks academic articles on the issues affecting the developers of educational systems and educators who implement and manage such<br />

systems. The articles should discuss the perspectives of both communities and their relation to each other:<br />

• Educators aim to use technology to enhance individual learning as well as to achieve widespread education and expect the technology to<br />

blend with their individual approach to instruction. However, most educators are not fully aware of the benefits that may be obtained by<br />

proactively harnessing the available technologies and how they might be able to influence further developments through systematic<br />

feedback and suggestions.<br />

• Educational system developers and artificial intelligence (AI) researchers are sometimes unaware of the needs and requirements of typical<br />

teachers, with a possible exception of those in the computer science domain. In transferring the notion of a 'user' from the humancomputer<br />

interaction studies and assigning it to the 'student', the educator's role as the 'implementer/ manager/ user' of the technology has<br />

been forgotten.<br />

The aim of the journal is to help them better understand each other's role in the overall process of education and how they may support<br />

each other. The articles should be original, unpublished, and not in consideration for publication elsewhere at the time of submission to<br />

Educational Technology & Society and three months thereafter.<br />

The scope of the journal is broad. Following list of topics is considered to be within the scope of the journal:<br />

Architectures for Educational Technology Systems, Computer-Mediated Communication, Cooperative/ Collaborative Learning and<br />

Environments, Cultural Issues in Educational System development, Didactic/ Pedagogical Issues and Teaching/Learning Strategies, Distance<br />

Education/Learning, Distance Learning Systems, Distributed Learning Environments, Educational Multimedia, Evaluation, Human-<br />

Computer Interface (HCI) Issues, Hypermedia Systems/ Applications, Intelligent Learning/ Tutoring Environments, Interactive Learning<br />

Environments, Learning by Doing, Methodologies for Development of Educational Technology Systems, Multimedia Systems/ Applications,<br />

Network-Based Learning Environments, Online Education, Simulations for Learning, Web Based Instruction/ Training<br />

Editors<br />

Kinshuk, Athabasca University, Canada; Demetrios G Sampson, University of Piraeus & ITI-CERTH, Greece; Ashok Patel, CAL<br />

Research & Software Engineering Centre, UK; Reinhard Oppermann, Fraunhofer Institut Angewandte Informationstechnik, Germany.<br />

Associate editors<br />

Alexandra I. Cristea, Technical University Eindhoven, The Netherlands; John Eklund, Access Australia Co-operative Multimedia<br />

Centre, Australia; Vladimir A Fomichov, K. E. Tsiolkovsky Russian State Tech Univ, Russia; Olga S Fomichova, Studio "Culture,<br />

Ecology, and Foreign Languages", Russia; Piet Kommers, University of Twente, The Netherlands; Chul-Hwan Lee, Inchon National<br />

University of Education, Korea; Brent Muirhead, University of Phoenix Online, USA; Erkki Sutinen, University of Joensuu, Finland;<br />

Vladimir Uskov, Bradley University, USA.<br />

Advisory board<br />

Ignacio Aedo, Universidad Carlos III de Madrid, Spain; Sherman Alpert, IBM T.J. Watson Research Center, USA; Alfred Bork,<br />

University of California, Irvine, USA; Rosa Maria Bottino, Consiglio Nazionale delle Ricerche, Italy; Mark Bullen, University of<br />

British Columbia, Canada; Tak-Wai Chan, National Central University, Taiwan; Nian-Shing Chen, National Sun Yat-sen University,<br />

Taiwan; Darina Dicheva, Winston-Salem State University, USA; Brian Garner, Deakin University, Australia; Roger Hartley, Leeds<br />

University, UK; Harald Haugen, Høgskolen Stord/Haugesund, Norway; J R Isaac, National Institute of Information Technology,<br />

India; Paul Kirschner, Open University of the Netherlands, The Netherlands; William Klemm, Texas A&M University, USA; Rob<br />

Koper, Open University of the Netherlands, The Netherlands; Ruddy Lelouche, Universite Laval, Canada; Rory McGreal, Athabasca<br />

University, Canada; David Merrill, Brigham Young University - Hawaii, USA; Marcelo Milrad, Växjö University, Sweden; Riichiro<br />

Mizoguchi, Osaka University, Japan; Hiroaki Ogata, Tokushima University, Japan; Toshio Okamoto, The University of Electro-<br />

Communications, Japan; Gilly Salmon, University of Leicester, United Kingdom; Timothy K. Shih, Tamkang University, Taiwan;<br />

Yoshiaki Shindo, Nippon Institute of Technology, Japan; Brian K. Smith, Pennsylvania State University, USA; J. Michael Spector,<br />

Florida State University, USA.<br />

Assistant Editors<br />

Sheng-Wen Hsieh, National Sun Yat-sen University, Taiwan; Taiyu Lin, Massey University, New Zealand; Kathleen Luchini,<br />

University of Michigan, USA; Dorota Mularczyk, Independent Researcher & Web Designer; Carmen Padrón Nápoles, Universidad<br />

Carlos III de Madrid, Spain; Ali Fawaz Shareef, Massey University, New Zealand; Jarkko Suhonen, University of Joensuu, Finland.<br />

Executive peer-reviewers<br />

http://www.ifets.info/<br />

Subscription Prices and Ordering Information<br />

Institutions: NZ$ 120 (~ US$ 75) per year (four issues) including postage and handling.<br />

Individuals (no school or libraries): NZ$ 100 (~ US$ 50) per year (four issues) including postage and handling.<br />

Single issues (individuals only): NZ$ 35 (~ US$ 18) including postage and handling.<br />

Subscription orders should be sent to The International Forum of Educational Technology & Society (IFETS), c/o Prof. Kinshuk, School of<br />

Computing & Information Systems, Athabasca University, 1 University Drive, Athabasca, Alberta T9S 3A3, Canada. Tel: +1 780 675 6812.<br />

Fax: +1 780 675 6148. E-mail: kinshuk@ieee.org.<br />

Advertisements<br />

Educational Technology & Society accepts advertisement of products and services of direct interest and usefulness to the readers of the<br />

journal, those involved in education and educational technology. Contact the editors at kinshuk@ieee.org.<br />

Abstracting and Indexing<br />

Educational Technology & Society is abstracted/indexed in Social Science Citation Index, Current Contents/Social & Behavioral Sciences,<br />

ISI Alerting Services, Social Scisearch, ACM Guide to Computing Literature, Australian DEST Register of Refereed Journals, Computing<br />

Reviews, DBLP, Educational Administration Abstracts, Educational Research Abstracts, Educational Technology Abstracts, Elsevier<br />

Bibliographic Databases, ERIC, Inspec, Technical Education & Training Abstracts, and VOCED.<br />

ISSN 1436-4522 1436-4522. (online) © International and 1176-3647 Forum (print). of Educational © International Technology Forum of & Educational Society (IFETS). Technology The authors & Society and (IFETS). the forum The jointly authors retain and the copyright forum jointly of the retain articles. the<br />

copyright Permission of the to make articles. digital Permission or hard copies to make of digital part or or all hard of this copies work of for part personal or all of or this classroom work for use personal is granted or without classroom fee use provided is granted that without copies are fee not provided made or that distributed copies<br />

are for not profit made or or commercial distributed advantage for profit and or commercial that copies advantage bear the full and citation that copies on the bear first the page. full Copyrights citation on the for components first page. Copyrights of this work for owned components by others of this than work IFETS owned must by be<br />

others honoured. than IFETS Abstracting must with be honoured. credit is permitted. Abstracting To with copy credit otherwise, is permitted. to republish, To copy to otherwise, post on servers, to republish, or to redistribute to post on to servers, lists, requires or to redistribute prior specific to lists, permission requires and/or prior a<br />

specific fee. Request permission permissions and/or a from fee. the Request editors permissions at kinshuk@massey.ac.nz.<br />

from the editors at kinshuk@ieee.org.<br />

i


Guidelines for authors<br />

Submissions are invited in the following categories:<br />

• Peer reviewed publications: a) Full length articles (4000 - 7000 words), b) Short articles, Critiques and Case studies (up to 3000 words)<br />

• Book reviews<br />

• Software reviews<br />

• Website reviews<br />

All peer review publications will be refereed in double-blind review process by at least two international reviewers with expertise in the<br />

relevant subject area. Book, Software and Website Reviews will not be reviewed, but the editors reserve the right to refuse or edit review.<br />

• Each peer review submission should have at least following items: title (up to 10 words), complete communication details of ALL<br />

authors , an informative abstract (75-200 words) presenting the main points of the paper and the author's conclusions, four - five<br />

descriptive keywords, main body of paper (in 10 point font), conclusion, references.<br />

• Submissions should be single spaced.<br />

• Footnotes and endnotes are not accepted, all such information should be included in main text.<br />

• The paragraphs should not be indented. There should be one line space between consecutive paragraphs.<br />

• There should be single space between full stop of previous sentence and first word of next sentence in a paragraph.<br />

• The keywords (just after the abstract) should be separated by comma, and each keyword phrase should have initial caps (for example,<br />

Internet based system, Distance learning).<br />

• Do not use 'underline' to highlight text. Use 'italic' instead.<br />

Headings<br />

Articles should be subdivided into unnumbered sections, using short, meaningful sub-headings. Please use only two level headings as far<br />

as possible. Use 'Heading 1' and 'Heading 2' styles of your word processor's template to indicate them. If that is not possible, use 12 point<br />

bold for first level headings and 10 point bold for second level heading. If you must use third level headings, use 10 point italic for this<br />

purpose. There should be one blank line after each heading and two blank lines before each heading (except when two headings are<br />

consecutive, there should be one blank like between them).<br />

Tables<br />

Tables should be included in the text at appropriate places and centered horizontally. Captions (maximum 6 to 8 words each) must be<br />

provided for every table (below the table) and must be referenced in the text.<br />

Figures<br />

Figures should be included in the text at appropriate places and centered horizontally. Captions (maximum 6 to 8 words each) must be<br />

provided for every figure (below the figure) and must be referenced in the text. The figures must NOT be larger than 500 pixels in width.<br />

Please also provide all figures separately (besides embedding them in the text).<br />

References<br />

• All references should be listed in alphabetical order at the end of the article under the heading 'References'.<br />

• All references must be cited in the article using "authors (year)" style e.g. Merrill & Twitchell (1994) or "(authors1, year1; authors2,<br />

year2)" style e.g. (Merrill, 1999; Kommers et al., 1997).<br />

• Do not use numbering style to cite the reference in the text e.g. "this was done in this way and was found successful [23]."<br />

• It is important to provide complete information in references. Please follow the patterns below:<br />

Journal article<br />

Laszlo, A. & Castro, K. (1995). Technology and values: Interactive learning environments for future generations. Educational Technology,<br />

35 (2), 7-13.<br />

Newspaper article<br />

Blunkett, D. (1998). Cash for Competence. Times Educational Supplement, July 24, 1998, 15.<br />

Or<br />

Clark, E. (1999). There'll never be enough bandwidth. Personal Computer World, July 26, 1999, retrieved July 7, 2004, from<br />

http://www.vnunet.co.uk/News/88174.<br />

Book (authored or edited)<br />

Brown, S. & McIntyre, D. (1993). Making sense of Teaching, Buckingham: Open University.<br />

Chapter in book/proceedings<br />

Malone, T. W. (1984). Toward a theory of intrinsically motivating instruction. In Walker, D. F. & Hess, R. D. (Eds.), Instructional<br />

software: principles and perspectives for design and use, California: Wadsworth Publishing Company, 68-95.<br />

Internet reference<br />

Fulton, J. C. (1996). Writing assignment as windows, not walls: enlivening unboundedness through boundaries, retrieved July 7, 2004,<br />

from http://leahi.kcc.hawaii.edu/org/tcc-conf96/fulton.html.<br />

Submission procedure<br />

Authors, submitting articles for a particular special issue, should send their submissions directly to the appropriate Guest Editor. Guest<br />

Editors will advise the authors regarding submission procedure for the final version.<br />

All submissions should be in electronic form. The editors will acknowledge the receipt of submission as soon as possible.<br />

The preferred formats for submission are Word document and RTF, but editors will try their best for other formats too. For figures, GIF<br />

and JPEG (JPG) are the preferred formats. Authors must supply separate figures in one of these formats besides embedding in text.<br />

Please provide following details with each submission: Author(s) full name(s) including title(s), Name of corresponding author, Job<br />

title(s), Organisation(s), Full contact details of ALL authors including email address, postal address, telephone and fax numbers.<br />

The submissions should be uploaded at http://www.ifets.info/ets_journal/upload.php. In case of difficulties, they can also be sent via<br />

email to (Subject: Submission for Educational Technology & Society journal): kinshuk@ieee.org. In the email, please state clearly that the<br />

manuscript is original material that has not been published, and is not being considered for publication elsewhere.<br />

ISSN 1436-4522 1436-4522. (online) © International and 1176-3647 Forum (print). of Educational © International Technology Forum of & Educational Society (IFETS). Technology The authors & Society and (IFETS). the forum The jointly authors retain and the copyright forum jointly of the retain articles. the<br />

copyright Permission of the to make articles. digital Permission or hard copies to make of digital part or or all hard of this copies work of for part personal or all of or this classroom work for use personal is granted or without classroom fee use provided is granted that without copies are fee not provided made or that distributed copies<br />

are for not profit made or or commercial distributed advantage for profit and or commercial that copies advantage bear the full and citation that copies on the bear first the page. full Copyrights citation on the for components first page. Copyrights of this work for owned components by others of this than work IFETS owned must by be<br />

others honoured. than IFETS Abstracting must with be honoured. credit is permitted. Abstracting To with copy credit otherwise, is permitted. to republish, To copy to otherwise, post on servers, to republish, or to redistribute to post on to servers, lists, requires or to redistribute prior specific to lists, permission requires and/or prior a<br />

specific fee. Request permission permissions and/or a from fee. the Request editors permissions at kinshuk@massey.ac.nz.<br />

from the editors at kinshuk@ieee.org.<br />

ii


Journal of Educational Technology & Society<br />

<strong>Volume</strong> 9 <strong>Number</strong> 4 <strong>2006</strong><br />

Table of contents<br />

Special issue articles<br />

Theme: E-learning and Human-Computer Interaction: Exploring Design Synergies for<br />

more Effective Learning Experiences<br />

Editorial: E-learning and Human-Computer Interaction: Exploring Design Synergies for more Effective<br />

Learning Experiences<br />

Alan Dix, Teresa Roselli and Erkki Sutinen<br />

Automatically Producing Accessible Learning Objects<br />

Angelo Di Iorio, Antonio Angelo Feliziani, Silvia Mirri, Paola Salomoni and Fabio Vitali<br />

A Boosting Approach to eContent Development for Learners with Special Needs<br />

Silvia Gabrielli, Valeria Mirabella, Stephen Kimani and Tiziana Catarci<br />

Supporting Students with a Personal Advisor<br />

Berardina De Carolis, Sebastiano Pizzutilo, Giovanni Cozzolongo, Pawel Drozda and Francesca<br />

Muci<br />

eLSE Methodology: a Systematic Approach to the e-Learning Systems Evaluation<br />

Rosa Lanzilotti, Carmelo Ardito, Maria F. Costabile and Antonella De Angeli<br />

Cooperative Project-based Learning in a Web-based Software Engineering Course<br />

Nicola Piccinini and Giuseppe Scollo<br />

Full length articles<br />

Discussion Tool Effects on Collaborative Learning and Social Network Structure<br />

Astrid Tomsic and Daniel D. Suthers<br />

Institutional Potential for Online Learning: A Hong Kong Case Study<br />

Sue L. Donoghue<br />

Reuse- and Aspect-Oriented Courseware Development<br />

Khaldoun Ateyeh and Peter C. Lockemann<br />

Using Online Education Technologies to Support Studio Instruction<br />

Diane M. Bender and Jon D. Vredevoogd<br />

The Effect of Video Presentation in a CBT Environment<br />

Ali Reza Montazemi<br />

Web-based Tools for Designing and Developing Teaching Materials for Integration of Information<br />

Technology into Instruction<br />

Kuo-En Chang, Yao-Ting Sung and Huei-Tse Hou<br />

Enhancing user support in open problem solving environments through Bayesian Network inference<br />

techniques<br />

Nikolaos Tselios, Adrian Stoica, Manolis Maragoudakis, Nikolaos Avouris and Vassilis Komis<br />

All Aboard - Destination Unknown: A Sociological Discussion of Online Learning<br />

Andra K. Goldberg and Frances Julia Riemer<br />

A Systemic Plan of Technology Integration<br />

Pi-Sui Hsu and Priya Sharma<br />

Surviving the shipwreck: what makes online students stay online and learn?<br />

Johannes C. Cronjé, Debbie E. Adendorff, Salome M. Meyer and Linda van Ryneveld<br />

Innovative Web-based Professional Development for Teachers of At-Risk Preschool Children<br />

Mable B. Kinzie, Stephen D. Whitaker, Kathy Neesen, Michael Kelley, Michael Matera and Robert<br />

C. Pianta<br />

ISSN 1436-4522 1436-4522. (online) © International and 1176-3647 Forum (print). of Educational © International Technology Forum & of Society Educational (IFETS). Technology The authors & Society and the (IFETS). forum The jointly authors retain and the the copyright forum jointly of the retain articles. the<br />

Permission copyright of to the make articles. digital Permission or hard copies to make of part digital or all or of hard this copies work for of part personal or all or of classroom this work use for is personal granted or without classroom fee provided use is granted that copies without are fee not provided made or that distributed copies<br />

are for profit not made or commercial or distributed advantage for profit and or that commercial copies bear advantage the full and citation that copies on the bear first page. the full Copyrights citation on for the components first page. Copyrights of this work for owned components by others of than this work IFETS owned must by be<br />

others honoured. than Abstracting IFETS must with be honoured. credit is permitted. Abstracting To with copy credit otherwise, is permitted. to republish, To copy to post otherwise, on servers, to republish, or to redistribute to post on to lists, servers, requires or to prior redistribute specific to permission lists, requires and/or prior a<br />

specific fee. Request permission permissions and/or from a fee. the Request editors permissions at kinshuk@massey.ac.nz.<br />

from the editors at kinshuk@ieee.org.<br />

1-2<br />

3-16<br />

17-26<br />

27-41<br />

42-53<br />

54-62<br />

63-77<br />

78-94<br />

95-113<br />

114-122<br />

123-138<br />

139-149<br />

150-165<br />

166-172<br />

173-184<br />

185-193<br />

194-204<br />

iii


How Does Educational Technology Benefit Humanity? Five Years of Evidence<br />

Pedro Hernández-Ramos<br />

Interoperability for Individual Learner Centred Accessibility for Web-based Educational Systems<br />

Liddy Nevile and Jutta Treviranus<br />

Reengineering Framework for Systems in Education<br />

Christophe Choquet and Alain Corbière<br />

Book review(s)<br />

Web-Based Intelligent e-Learning Systems: Technologies and Applications<br />

Reviewer: Charalampos Karagiannidis<br />

Visual Media and the Humanities: A Pedagogy of Representation<br />

Reviewer: Lauren Glenn<br />

ISSN 1436-4522 1436-4522. (online) © International and 1176-3647 Forum (print). of Educational © International Technology Forum of & Educational Society (IFETS). Technology The authors & Society and (IFETS). the forum The jointly authors retain and the copyright forum jointly of the retain articles. the<br />

copyright Permission of the to make articles. digital Permission or hard copies to make of digital part or or all hard of this copies work of for part personal or all of or this classroom work for use personal is granted or without classroom fee use provided is granted that without copies are fee not provided made or that distributed copies<br />

are for not profit made or or commercial distributed advantage for profit and or commercial that copies advantage bear the full and citation that copies on the bear first the page. full Copyrights citation on the for components first page. Copyrights of this work for owned components by others of this than work IFETS owned must by be<br />

others honoured. than IFETS Abstracting must with be honoured. credit is permitted. Abstracting To with copy credit otherwise, is permitted. to republish, To copy to otherwise, post on servers, to republish, or to redistribute to post on to servers, lists, requires or to redistribute prior specific to lists, permission requires and/or prior a<br />

specific fee. Request permission permissions and/or a from fee. the Request editors permissions at kinshuk@massey.ac.nz.<br />

from the editors at kinshuk@ieee.org.<br />

205-214<br />

215-227<br />

228-241<br />

242-243<br />

244-245<br />

iv


Dix, A., Roselli, T., & Sutinen, E. (<strong>2006</strong>). E-learning and Human-Computer Interaction: Exploring Design Synergies for<br />

more Effective Learning Experiences. Educational Technology & Society, 9 (4), 1-2.<br />

E-learning and Human-Computer Interaction: Exploring Design Synergies<br />

for more Effective Learning Experiences<br />

Alan Dix<br />

Computing Department, InfoLab21, South Drive, Lancaster University, LA1 4WA, Lancaster, UK<br />

alan@hcibook.com<br />

Teresa Roselli<br />

Department of Computer Science, University of Bari, Via E. Orabona 4, 70125 Bari, Italy<br />

roselli@di.uniba.it<br />

Erkki Sutinen<br />

Department of Computer Science, University of Joensuu, 80110 Joensuu, Finland<br />

erkki.sutinen@cs.joensuu.fi<br />

The current trend in developing e-learning systems is largely empirical anecdotal, while consolidated, evidencebased<br />

models ensuring systematic and pedagogically sound learning experiences are still lacking. Up to now, the<br />

e-learning community has mainly focused on investigating the technical qualities of such systems, but has tended<br />

rather to neglect their didactic effectiveness and usability. Thus the e-learning community needs to devise and<br />

discuss new criteria for the design of more usable and innovative systems supporting creative learning, based on<br />

strategies which can, on the one hand, guide the learner to make the most effective use of the didactic content,<br />

and, on the other hand, refrain from being too intrusive in scaffolding the learning process.<br />

A major challenge currently faced by e-learning systems’ designers is the development of improved tools better<br />

able to engage novice learners and sustain their online learning activities any time and anywhere. Human-<br />

Computer Interaction (HCI) theories and methodologies can support the design of appropriate e-learning settings<br />

responding to the complex and rapidly changing requirements of both the academic and business contexts of our<br />

society. Basically, e-learning applications should become smart enough to adapt themselves to the students’<br />

learning styles and to assure high standards of accessibility and usability, in order to make learners’ interaction<br />

with the systems as natural and intuitive as possible.<br />

Stronger synergies should be established between the design of e-learning experiences and the analysis of<br />

learners’ preferred interactions with e-learning environments. To reach this objective an evolving learner-centred<br />

design perspective should be adopted, taking into account also the typical learning styles shared within the<br />

different cultural contexts. Future studies based on these assumptions could provide valuable results and inspire<br />

interesting lines of thought for the intersection of HCI and e-learning.<br />

To be successful, the synergy approach requires that the researchers design new tools for the users whose<br />

feedback from concrete user scenarios is analyzed from the very beginning of and throughout the design process.<br />

In this regard, special needs education provides researchers with a particularly beneficial context since diverse<br />

learners force the designers to really listen to the users feedback to be able to create functional e-learning tools.<br />

This special issue features the best papers presented at the workshop e-learning and Human-Computer<br />

Interaction: Exploring Design Synergies for more Effective Learning Experiences part of the International<br />

Conference on Human-Computer Interaction (INTERACT 2005) held on Rome in September 2005.<br />

The workshop aimed at stimulating discussions about the latest advances in e-learning, based on application of<br />

HCI approaches to distant education.<br />

The overall acceptance rate of the two blind review processes was 30%. The five published papers cover several<br />

key themes in the e-learning and HCI research area.<br />

There are two papers in the area of accessibility of didactic resources. Di Iorio, Feliziani, Mirri, Salomoni and<br />

Vitali present a learning object creation and management process based on common personal productivity tools,<br />

which guarantees both content accessibility as well as universality and offers a simple and friendly interface to<br />

authors. Gabrielli, Mirabella, Kimani and Catarci propose a design method for increasing the quality of elearning<br />

materials for learners with special needs and an authoring environment to support authors in their<br />

development of didactic material matching those needs.<br />

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the<br />

copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies<br />

are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by<br />

others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior<br />

specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.<br />

1


Adaptivity is one of the most important topics of e-learning research. The aim is to supply more and more<br />

customized learning paths in order to meet the learners' needs and to achieve more effective learning. De Carolis,<br />

Pizzutilo, Cozzolongo, Drozda and Muci present an architecture of an Embodied Conversational Agent (ECA)<br />

designed to assist students by providing personalized suggestions related both to the fruition of didactic material<br />

and to general orientation for student’s daily life.<br />

Learner needs are the starting point of the Lanzilotti, Ardito, Costabile and De Angeli paper. The authors<br />

highlight the lack of high-quality systems tailored to the needs of individual users and groups. They refine the<br />

concept of quality of e-learning systems and propose a framework TICS (Technology, Interaction, Content,<br />

Services), which focuses on the user-system interaction as one of the most important aspects to be considered<br />

when designing or evaluating an e-learning system. Moreover they propose an evaluation methodology called<br />

eLSE (e-Learning Systematic Evaluation).<br />

The last paper Piccinni and Scollo presents and analyzes a case study in software engineering education,<br />

spanning over a seven-year evolution, characterized by a blend of educational techniques: traditional classroom<br />

lectures, textbook and lecture notes, as well as a web-based cooperation platform, supporting interaction and<br />

self-organization of laboratory projects.<br />

Conclusions<br />

The special issue and workshop have played a major, "prime mover" role in fostering a greater sense among<br />

HCI-oriented researchers of e-learning and tracking important directions in e-learning research for the coming<br />

years. As Guest Editors we hope that this special issue will provide an overview of studies highlighting the<br />

multiple relationships between technological and educational approaches to the design of e-learning<br />

environments.<br />

Acknowledgements<br />

Special thanks to our reviewers for the relevant and detailed comments provided to the authors of all papers<br />

submitted. We would like to thank all conference chairs, the reviewers, all the members of the workshop<br />

secretariat and all those who helped to make the workshop and this resultant special issue a success.<br />

2


Di Iorio, A., Feliziani, A. A., Mirri, S., Salomoni, P., & Vitali, F. (<strong>2006</strong>). Automatically Producing Accessible Learning<br />

Objects. Educational Technology & Society, 9 (4), 3-16.<br />

Automatically Producing Accessible Learning Objects<br />

Angelo Di Iorio, Antonio Angelo Feliziani, Silvia Mirri, Paola Salomoni and Fabio<br />

Vitali<br />

Department of Computer Science, Università di Bologna, Via Mura Anteo Zamboni 7, 40126 Bologna BO, Italy<br />

diiorio@cs.unibo.it<br />

afelizia@cs.unibo.it<br />

mirri@cs.unibo.it<br />

salomoni@cs.unibo.it<br />

vitali@cs.unibo.it<br />

ABSTRACT<br />

The “Anywhere, Anytime, Anyway” slogan is frequently associated to e-learning with the aim to emphasize<br />

the wide access offered by on-line education. Otherwise, learning materials are currently created to be used<br />

with a specific technology or configuration, leaving out from the virtual classroom students who have<br />

limited access capabilities and, in particular, students with special needs. On the contrary, accessibility of<br />

learning materials is a key issue to ensure a whole inclusion of people with disability in instructional<br />

process, and, consequently, to prevent risks of “digital exclusion”. The foremost explanation for the<br />

creation of not inclusive materials is the assumption that creating accessible and universal Learning Objects<br />

(LO) is a dismaying and lengthy task, partially supported by complicated ad-hoc tools. New simple<br />

mechanisms that drive authors in creating accessible LO are needed to enlarge LO audience; they should<br />

contemporary support students’ needs (while reading) and reduce authors’ efforts (while writing). In this<br />

paper we present a LO creation and management process, based on common personal productivity tools,<br />

which guarantees both content accessibility as well as universality and offers a simple and friendly interface<br />

to authors.<br />

Keywords<br />

Accessibility, Authoring, Standards compliance, Content creation<br />

Introduction<br />

Tools ease of use for e-learning contents creation has been always paired with the continuously raising<br />

expectations about the quality of the final result. Although nowadays many sophisticated tools to create elearning<br />

contents can be track down, authors’ job is increasingly complicated by new quality requirements. Two<br />

main principles have contemporaneously to be addressed, which conjure to make e-learning authors’ job once<br />

again difficult and full of technical issues to be tackled:<br />

1. Accessibility: producing fully barrier-free learning contents is one of the key issues to meet the goal of an<br />

inclusive “knowledge society”. Accessibility, with full and authoritative backing from standard bodies such<br />

as the W3C and national governments through laws and norms, further raise the bar for web authors.<br />

2. Portability: conformance to e-learning standards enhances contents portability and is recognized as one of<br />

the fundamental aspects to preserve contents value. Specific e-learning standards guarantee that compliant<br />

contents could be used in several learning systems.<br />

Existing tools, because of several different reasons, still fall short in providing a full authoring environment, both<br />

capable of dealing with all the appropriate technical aspects and, at the same time, as easy to use as normal<br />

desktop applications such as word processors or presentation applications. As a consequence of such a lack, an<br />

additional professional figure is required to step in between the educators and the final e-learning system, which<br />

assists the authors in creating final learning objects t satisfying all modern technical requirements.<br />

How should the ideal authoring tool be, in order to fully assist educators in creating accessible content for<br />

modern Learning Management Systems? What kind of support could we build for users that do not really want<br />

to deal with all the technical details of current e-learning technologies? We have identified at least seven<br />

dimensions:<br />

1. Ease of use: the tool should be at least as easy to use as existing word processors and presentation tools. In<br />

particular, it should behave as similar as possible to their work so as to lower its learning curve.<br />

2. Ease of re-use: the tool should help and assist authors in reusing and converting documents and materials<br />

they have already prepared in past times by using any of a large number of desktop tools, to the new elearning<br />

platforms.<br />

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain<br />

the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that<br />

copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work<br />

owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists,<br />

requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.<br />

3


3. Ease of editing and updating: whenever the authoring process starts with a conversion step, it oftentimes<br />

becomes a one-way process that cannot be repeated at will, but it costs money and time and requires<br />

expertise to be done several times. The conversion process raises difficulties in regularly editing and<br />

updating the published material, and constrains users to directly edit the converted material rather than the<br />

source documents which were originally used before the conversion.<br />

4. Standards support: the tool should generate learning objects that can be read by a large variety of<br />

commercial and open source e-learning platforms. It practically means that such a tool should produce<br />

learning objects according to some major e-learning standards, defined by different standardization<br />

organization.<br />

5. Visual Homogeneity: the tool should produce content that easily undergoes platform- and site-specific styles<br />

and look&feel, by fully and easily adapting any content to the templating and styling locally mechanism<br />

adopted.<br />

6. Universality: the tool should generate content which under appropriate conditions (e.g., the choice of<br />

different and independent templates), can be fully and at best quality displayed on a wide variety of<br />

applications, including non-dominant versions of browsers and operating systems, older versions of<br />

browsers and operating systems, new and emerging hardware devices (such as PDAs, cellular phones,<br />

interactive TV sets, etc.).<br />

7. Accessibility: the tool should create fully accessible content according to international standards and<br />

national laws. It should fully assist the author, silently generating the accessible structures that can be<br />

automatically deduced from existing content, and prompting the author in providing anything cannot be<br />

automatically generated (e.g., the textual description of images).<br />

Accessibility is of paramount importance, especially when dealing with e-learning applications: learners with<br />

disabilities can benefit a great advantage from e-learning, not simply because it allows distant and flexible<br />

learning activities, but mainly because it could support impaired students in overcoming barriers to resources<br />

which would otherwise be hard to access (Salomoni et al., 2004). In particular, learners with difficulties in<br />

accessing to printed materials (i.e. people with visual impairments) can take a great advantage of the integration<br />

of digital materials into teaching practice. Across the world, laws are in place or under definition to ensure that<br />

interactive/on-line services and, sometimes, specifically e-learning, are made accessible to citizen with<br />

disabilities (Italian Parliament, 2004; U.S. Rehabilitation Act Amendments of 1998, 1998). Two main strategies,<br />

combined together, are used in order to meet accessible requirements:<br />

Appropriate use of W3C standard compliant technologies, in order to enhance interoperability and<br />

portability (World Wide Web Consortium, 2000; World Wide Web Consortium, 1999b; World Wide Web<br />

Consortium, 2001b).<br />

Use of alternative versions of inaccessible content, in order to accommodate different users, e.g., students<br />

using limited or obsolete devices and/or students with sensorial impairments (IMS Global Learning<br />

Consortium, 2002b; World Wide Web Consortium, 1999a; World Wide Web Consortium, 1999c).<br />

In this paper we present a methodology and some tools for the creation and management of accessible and<br />

universal learning objects (LO) which considerably improves the process across the seven mentioned<br />

dimensions. Our methodology involves different tools, from widespread text editors, which are used by authors<br />

to easily produce content, till an ad-hoc application, called ISA-BeL, designed and implemented to support<br />

automatic production of standard compliant e-learning materials. The process has been widely used to publish<br />

about 40 learning modules which are currently in use by our University for several e-learning activities in a<br />

number of subjects. Accessibility of the whole process and all the produced LOs have been verified on the field.<br />

The remainder of the paper is organized as follows. Section 2 provides some background information about<br />

authoring e-learning content platform. Section 3 introduces the creation and management process in e-learning<br />

contexts using ISA-BeL, while section 4 fully details a use-case in producing accessible and standard e-learning<br />

materials by using ISA-BeL. Finally, Section 5 provides some conclusions and suggestions for future works.<br />

Background<br />

Systems and standards<br />

Many recent efforts in the field of knowledge management address e-learning, i.e., distance learning based on<br />

the use of personal computers. Systems providing e-learning services can be divided in two main categories:<br />

LMSs (Learning Management Systems), which are web-based platforms by actually providing content to the<br />

users and LCMSs (Learning Content Management System), the authoring environments used to create learning<br />

4


objects. The main features of an LCMS are related to the content management, from the production to the<br />

storage including reusability and distribution of content. On the other hand an LMS manages the administrative<br />

functions (i.e. users’ activities logging and curriculum management), the distribution of contents to learners, and<br />

the tracking of the learners’ experiences and assessments.<br />

A relevant role is played by existing e-learning standards, in ensuring interoperability and reuse of didactical<br />

materials. Main interoperability specifications have been developed by IEEE (Institute of Electrical and<br />

Electronics Engineers), with a specific working group, the Learning Technology Standards Committee, which is<br />

working on e-learning standardization (IEEE LTSC WG12, <strong>2006</strong>) and IMS (Instructional Management System)<br />

Global Learning Consortium (IMS Global Learning Consortium, <strong>2006</strong>). A relevant role is also played by<br />

Advanced Distributed Learning (ADL) initiative (Advanced Distributed Learning, <strong>2006</strong>), which has developed a<br />

de-facto standard called SCORM (Shareable Content Object Reference Model) (Advanced Distributed Learning,<br />

2004c), based on some specifications previously defined by IEEE-LTSC and IMS. It is structured in modules<br />

defining sub specifications. In this context let us introduce two SCORM modules:<br />

1. The Content Aggregation Model (CAM) defining the structure of learning material, and describes it with<br />

metadata (Advanced Distributed Learning, 2004a).<br />

2. The Run-time Environment (RTE), a JavaScript Application Programming Interface (API) able to deliver<br />

real time information about user actions (Advanced Distributed Learning, 2004b) to a LMS or a LCMS.<br />

Learning Object Production<br />

An interesting field in e-learning research is the simplification and automation of the learning objects production:<br />

some works can be found in literature which are devoted to automatically produce LOs starting from available<br />

didactical materials (created in many different formats), or by directly generating SCORM-compliant contents.<br />

Several projects and products provide authoring tools for e-learning materials (4system, <strong>2006</strong>; Macromedia,<br />

<strong>2006</strong>; ReadyGo Inc., <strong>2006</strong>; SumTotal Systems Inc., <strong>2006</strong>). They commonly imply high-level technical skills in<br />

order to be appropriately used by authors, which have typically to complete a preliminary training phase to<br />

effectively use them (as they can do with well-known “productivity tools”) These products generally provide<br />

authors with different interfaces and functionalities so that they can create e-learning materials, manage<br />

resources, aggregate contents, add meta-data, and so on. Some of these tools are free, while some others are<br />

commercial products. In many cases, the use of these new tools cannot be fully appreciated by authors of<br />

e-learning materials, who may prefer to rely on well-known productivity tools, which could allow them some<br />

savings in time and money. Products and platforms are designed by moving onto in this direction and they<br />

generate e-learning course materials starting from well-known productivity tools, such as Microsoft Word<br />

(Horizon Wimba, <strong>2006</strong>; Serco, <strong>2006</strong>). The main advantage in exploiting such products is that no learning and<br />

training phases are needed. On one hand these tools provide a too rigid structure in drawing up created contents,<br />

on the other hand they keep authors’ stylistic choices, instead of maintaining only designers’ ones, invalidating<br />

accessibility and usability principles. In addition, they do not support useful mechanisms to manage DTD<br />

configurations or allow the insertion of new structure style. One of these products (Horizon Wimba, <strong>2006</strong>)<br />

provides a partial support to accessibility of created contents, but, in some cases, generated LOs are not<br />

compliant to international guidelines and laws, denying actual benefits to learners with disabilities.<br />

Another interesting issue in e-learning course contents creation is the design o Generative Learning Objects<br />

(GLOs) (Boyle et al., 2004; Bradley et al., 2004; Morales, 2005). The underlying idea is based on the division<br />

into two different parts the LOs creation. The first one consists of building a Learning Object Template (LOT),<br />

while the second one is devoted to adding the template a subject specific content. The LOT encloses the deep<br />

general structure of the e-learning course. Once a template has been created, authors and/or tutors can add<br />

different subject specific contents, i.e. the surface structure, so as to produce Learning Objects which fit the<br />

specific fields of the discipline.<br />

Finally, let us consider a new way to think intermediate data format that is promising to have an interesting<br />

impact on LO production. Microformats (Microformats.org, <strong>2006</strong>). Microformats are a set of simple open data<br />

format standard which are developed and implemented for more/better structured web microcontent publishing.<br />

Some already developed microformats specify for example formats for calendar and events data. Many other<br />

microformats are currently in a developing phase. In (Downes, <strong>2006</strong>) authors propose Microformats use in elearning,<br />

by conceiving it as a network phenomenon, so as to facilitate a personal e-learning centre design.<br />

5


Accessible and personalized Learning Objects<br />

Other academic projects and works are devoted to produce accessible e-learning contents and/or to personalize<br />

them. One of these proposes the design of a prototype (Gabrielli et al., 2005; Mirabella et al., 2004) which drives<br />

authors in creating accessible didactical materials. The authoring interface of this prototype is developed in Java<br />

and its main aim is to support authors’ job with suggestions and examples. They are provided in order to show<br />

and explain how accessible contents have to be produced by providing appropriate and equivalent alternatives to<br />

learners and according to W3C Web Content Accessibility Guidelines (World Wide Web Consortium, 1999c). In<br />

(Kelly et al., 2005), the authors propose a holistic approach for addressing e-learning accessibility which takes<br />

into account e-learning usability of pedagogic issues, student learning styles and needs and some other<br />

significant constrains which usually accessibility guidelines do not cover. The main aim of this work is showing<br />

how the whole quality of e-learning contents can be assured by using this approach.<br />

Open Adaptive Learning Environment (OPAL) 0 is an e-learning service, which considers the personalization of<br />

e-learning course contents and is based on the delivery of personalized contents. The personalization is done on<br />

the basis of students’ cognitive and presentation learning preferences by using aggregation models based on<br />

SCORM (Advanced Distributed Learning, 2004c). Another interesting academic project is ELENA (Dolog et al.,<br />

2003; Dolog et al., 2004), which supports personalized access to distributed learning repositories. The approach<br />

to customization employed in this project takes advantage of semantic Web technologies and metadata<br />

description standards, such as LOM (IEEE LTSC WG12, 2002) and IMS AccessForAll Meta-data (IMS Global<br />

Learning Consortium, 2002a). In addition, it adapts and customizes access, delivery and consuming of learning<br />

services and LOs on the basis of rule-based matching of contents and learners descriptions.<br />

No one of the above described projects and products combines accessibility issues with methodologies to<br />

automatically produce LOs, like the ones described in Section 2.2.<br />

From content creation to e-learning delivery<br />

Context: the A 3 Project<br />

Teaching basic computer knowledge is becoming a matter of big interest in a lot of fields, particularly in<br />

Universities, where in every degree course is necessary to certify a minimal skill in computer knowledge. For<br />

this reason, the Department of Computer Science at the University of Bologna has developed a project (called<br />

A 3 , Accessible Learning Environment (University of Bologna, 2004), “Ambiente Accessibile d’Apprendimento”<br />

in Italian language) for the creation and fruition of contents taking in a particular account the training structure<br />

uniformity, a low management cost, and a little effort in resources and time consumption for the process startup<br />

Main requirements related to LOs used in A³ are due to:<br />

Accessibility and Web standard compliance. The project was developed and carried out inside an Italian<br />

University and it respects the Italian Law on Information Accessibility, the so called “Stanca Act” (Italian<br />

Parliament, 2004). In particular, as requested by the Italian Law about Web accessibility: (i) all templates<br />

completely separate presentation from content and structure, by producing XHTML Strict 1.0 and CSS 2.0<br />

code, (ii) layout and font dimensions are relative, and (iii) foreground and background colors are compliant<br />

to W3C requirements.<br />

Portability of LOs and e-learning standard compliance. Learning objects produced in A³ are package<br />

SCORM 1.2 RTE compliant (Advanced Distributed Learning, 2004b), so that contents can be imported in<br />

every LMS (Learning Management System) SCORM compliant.<br />

The Italian Law about accessibility (Italian Parliament, 2004), analogously to the WCAG 1.0 (World Wide Web<br />

Consortium, 1999c) requires contents and services to be provided independently from the use of scripts and<br />

programming objects. This means that a complete use of SCORM is not possible in respect of the current set of<br />

Italian (or W3C) requirements, and specifically, only the SCORM-CAM (Advanced Distributed Learning,<br />

2004a) standard has a complete compatibility with accessibility specification. Otherwise the SCORM-RTE<br />

(Advanced Distributed Learning, 2004b) module, devoted to Real Time activities, is completely based on the use<br />

of Javascript and it is not compatible with the (current) accessibility requirements. Let us note that both the<br />

Italian Law (near to adopt specific requirements on e-learning) and the WCAG (near to version 2.0) will be<br />

updated in the next future to allow Javascript and it will permit a full use of SCORM (World Wide Web<br />

Consortium, <strong>2006</strong>).<br />

6


Architecture<br />

Producing usable, accessible and universal content is a complex and time-consuming task that, in most cases,<br />

still requires expertise to the authors. All constrains derived from A³ requirements presented in the previous<br />

section exacerbate this condition. The basic idea to overcome authors difficulties, which is proposed by A³ was<br />

based on developing a chain of tools, called ISA-BeL (Di Iorio et al., 2005), which allows users to easily create<br />

accessible and portable learning objects (LOs) by mainly using a word processor (or, more generally, a<br />

productivity tool). Thus, ISA-BeL does not require authors to know technical details of e-learning technologies<br />

and languages, to master complex LCMSs or to use potentially complicated ad-hoc tools. ISA-BeL author writes<br />

a document of raw content and he/she indicates the role of each fragment (by using styles according to a set of<br />

given guidelines), and the conversion engine transforms each fragment in a proper element of the final learning<br />

object. Actually, information about the whole output structure as well as some metadata are required, but<br />

inserting such data is really simple and fast, as we will discuss later.<br />

A three step workflow can be now defined, from knowledge creation to e-learning provision:<br />

1. Authoring (Content creation), done by teachers using a word processor (or alternatively a different personal<br />

productivity tool, such as a presentation application).The output of this phase is a set of documents in<br />

common formats like rtf, doc, ppt, sxw, etc. containing accessible learning materials.<br />

2. Producing (Content transformation), i.e. the process creating a LO from a set of documents produced during<br />

phase 1. The output of this phase is a LO which has to maintain accessibility features embedded in original<br />

documents.<br />

3. Delivery (Content distribution), the real e-learning service, provided by a LMS which guests the LO which<br />

is produced in phase 2. The LMS has to guarantee accessibility of content and service provisioning.<br />

The whole process is depicted in Figure 1, which also shows the content production step performed by ISA-BeL<br />

and described in the Section 3.4 with more details.<br />

The output of ISA-BeL is not a simple set of common HTML pages, but a group of several alternative contents,<br />

which are used to enhance portability and accessibility, such as:<br />

1. Learning Objects, compliant to SCORM-CAM 1.2 or 1.3 (Advanced Distributed Learning, 2004a); this<br />

version can be uploaded on a SCORM compliant LMS and provided to students throughout the Web.<br />

2. Web-based materials, to be distributed on-line on the Web or on an optical storage device (CD or DVD) or<br />

to be delivered through the Internet.<br />

3. Printed materials in PDF format, obtained from the original contents by using XSL Formatting Objects<br />

(World Wide Web Consortium, 2001a).<br />

Authoring<br />

.doc<br />

Word<br />

Processor<br />

.htm<br />

ISA<br />

.xml<br />

BeL<br />

ISA-BeL<br />

Web<br />

Contents<br />

Figure 1. Authoring-management-provision of accessible e-learning by means of ISA-BeL<br />

The ISA-BeL support for content creation consists of providing users an alternative and simplified way to<br />

express all the data and content useful to create learning objects. Focusing on the resources packaging (even if<br />

the whole project we are working on currently supports tracking, run-time monitoring and assessments<br />

management too), a learning object can be defined as a set of structured resources supplied with a (SCORM)<br />

manifest that describes them.<br />

LO<br />

PDF<br />

LMS<br />

HTTP<br />

server<br />

AUTHORING PRODUCTION DELIVERY<br />

7


From this definition, we figured out alternative mechanisms to (i) indicate which pages compose the learning<br />

object and which content elements compose each page, (ii) verify these content elements express all the required<br />

information and (iii) add metadata associated to the learning object. We have provided many facilities to sustain<br />

contents creation also by authors which are non-expert in technologies and standards concerning accessibility<br />

and usability issues. In particular for accessibility, we supply authors with MSWord styles which are accessible<br />

with a custom toolbar. Each style is meant for a specific role in the final document; the role could regard a<br />

particular required accessibility constraint or a predefined semantic position in the text. The correct use of styles<br />

according to their implicit semantics is managed by some macros which alert the author about lacks of<br />

information. Moreover, the author can use a Word embedded form to fill in specific metadata.<br />

An ISA-BeL author, in fact, writes his/her contents by using a word processor, and he/she assisted by three<br />

different tools, specifically designed to cover these three aspects:<br />

An authoring toolbar which offers a fast access to him/her main activities such as defining presentational<br />

aspects and structural elements, inserting accessibility related information and so on. A screenshot of the<br />

ISA-BeL toolbar is depicted in the following Figure 2.<br />

A verification toolbar which runs automatic controls over the respect of accessibility and universality<br />

constrains. For example, some exploited controls are: have any image an alternative description? Has any<br />

table a summary? Are titles used according to the correct hierarchical structure? Other manual controls are<br />

made by the post-production team (about foreign language words, acronyms, abbreviations, and so on); if<br />

the LO does not respect such constrains then authors have to correct and control contents again, before<br />

starting the production phase.<br />

A set of forms to collect SCORM Metadata. As expected, a key role is played by the SCORM metadata:<br />

while some of them can be derived by the system (last-saved date, file size, language, version number and<br />

so on), other information have necessarily to be provided by the authors.<br />

A step-by-step description of author activities, from creation to production is detailed in the following Section 4.<br />

Production<br />

Figure 2. The ISA-BeL authoring toolbar (in MS Word)<br />

The production process is performed by an ad-hoc application, ISA-BeL which is composed of two modules:<br />

ISA (Immediate Site Activator): a conversion tool which actually transforms document from the word<br />

processor format into an intermediate XML representation, enriched by all the necessary metadata. An<br />

introduction about ISA architecture and functionalities can be found in (Vitali, 2003).<br />

BeL (Backed e-Learning): a stand-alone application which gathers all the information stored in the<br />

intermediate XMLs, creates the SCORM structures (in particular the tracking scripts and the manifest file)<br />

and merges the content into a single .ZIP file, by processing the output of ISA. BeL also integrates into the<br />

LO a (multimedia) recorded accessible video lecture, which is automatically transcoded through a different<br />

line of the LO production. More information about this transcoding process can be found in (Salomoni et al.,<br />

2005).<br />

The production process is based on a set of templates and configuration files which are used to define structural<br />

aspects as well as layout and graphical aspects of the automatically produced LO.<br />

Delivery<br />

The system has been used for the above mentioned A 3 project. Contents which were produced by the automatic<br />

process are loaded on the e-learning platform by means of standard (for SCORM compliant contents) import<br />

procedures. Our choice has been made between the large number of open-source platforms and after a deep<br />

8


testing period we have adopt ATutor platform (Adaptive Technology Resource Centre, University of Toronto,<br />

<strong>2006</strong>). The choice has been driven by the built-in usability support even though it has been necessary modifying<br />

the platform to adapt it to the Italian Law on accessibility (Italian Parliament, 2004) which is narrower than the<br />

Canadian one. Accessibility of the adopted LMS partially guarantees that produced contents maintain<br />

accessibility features and ensures accessibility and portability of services (chat, forum, news). Some<br />

modifications were needed to completely meet the “Stanca Act” requirements, and particularly the constraint to<br />

use Strict (X)HTML code.<br />

The main delivery of A³ was through the LCMS, but ISA-BeL produces also a printable version and a fully<br />

HTML one (which could be distributed both on the Internet and on a CD/DVD).<br />

Simply creating an accessible LO<br />

This section offers a detailed description of creation and production phases, by presenting a step-by-step<br />

description of author activities. Let us consider a specific scenario inside the A³ Project, in which the author uses<br />

MS Word to create contents that are written in Italian to introduce some basic IT skills. In particular, our author<br />

has to structure a lecture on “Digital Documents” through 6 sub-modules about different aspects of the main<br />

topic. Each sub-module is created by generating a .doc file which includes its contents. In order to complete the<br />

LO production, the author has to:<br />

Prepare the 6 files, by creating the sub-modules content with MS Word. The module is completed with some<br />

other .doc files containing scope, goals and references.<br />

Verify the structure of the produced contents, file by file.<br />

Add the SCORM Metadata.<br />

Obtain the .htm version of each file, by using the “Save as” option of the “File” menu which is provided by<br />

MS Word.<br />

Use the ISA online tool to produce the set of XML intermediate files.<br />

Use the BeL online tool to compose and produce an accessible LO.<br />

This sequence of activities is fully described in the following.<br />

Creating contents<br />

Let us suppose to work on the sub-module about “Digital Images” which starts with a general description of<br />

vector and bitmap formats. At the beginning of his/her work our author writes the title of the first page of the<br />

section (in Italian “Bitmap e Vettoriale”) and uses the “Title 1” style to format it. The page is based on a short<br />

explanation text describing this specific topic and the author uses the authoring toolbar to add a suitable format<br />

to it. In order to complete his/her exposition, the author adds a figure showing a bitmap and a vector image and<br />

compares the two data representations. Such an image has to be formatted by using the “image” style, available<br />

through the toolbar. The image needs a short description that will be used as a caption and a longer explanation<br />

which completely describes the information embedded inside the image. This description will be provided to<br />

blind students and to students that access to didactical contents through a non visual browser. Both the<br />

descriptions are identified by using the appropriate styles, and specifically:<br />

the “short description” style, associated to a button on the toolbar, and<br />

the “long description”style, associated to another button on the toolbar.<br />

The style “Short description” automatically appears after each use of the style “Image” in order to suggest the<br />

author a correct use of styles. Analogously, the style “Long description” appears after each use of the style<br />

“Short description”.<br />

The page is closed by opening a new one, e.g. inserting a section break; this task could be performed by using<br />

the appropriate MS Word menu option or by activating a related button. The final result is depicted in the<br />

following Figure 3, showing the final layout of the page inside the MS Word editor.<br />

The author completes the sub-module by creating a sequence of pages, similarly to that one described above.<br />

Appropriate mechanisms to format and structure contents are also provided to insert tables, terms in foreign<br />

languages, animations and all the other elements that can be included in an A³ module.<br />

9


Verifying contents<br />

Figure 3. The page created with MS Word<br />

Some of the actions due to obtain a universal and accessible content could be forgotten by the author that<br />

sometimes prefers to write a fast draft and improve it by adding additional information later on. The system<br />

provides a verification toolbar, depicted in the following Figure 4, to support the author in controlling the<br />

correctness and completeness of his contents.<br />

Figure 4. The verification toolbar<br />

With the verification toolbar, the author can control the syntactical respect of all the guidelines which guarantee<br />

universality and accessibility of the learning materials. A typical control verifies the presence of the required<br />

description of images; in this case, two different alerts are provided (as depicted in Figure 5):<br />

An error, every time a short description is missing. The short description corresponds to the alt attribute of<br />

the img tag and is required to guarantee the accessibility of graphical information, thus the absence of a<br />

short description it is considered as a heavy error (World Wide Web Consortium, 1999c).<br />

A warning, related to every missing long description. The long description is needed for complex graphical<br />

contents (e.g. charts, graphs, etc.), which need additional descriptive information to be understood by blind<br />

people or by people accessing with no-graphical browsers (World Wide Web Consortium, 1999c), e.g. Lynx<br />

(Lynx Project, <strong>2006</strong>).<br />

Figure 5. Alert messages<br />

The automatic control can catch only syntactical errors, such as a missing description. It is substantially<br />

impossible to automatically ensure the description will be adequate to the image. Other constraints to the<br />

structure and the accessibility of contents could be activated through the verification toolbar such as the<br />

hierarchical use of nested headings and the presence of summaries for data tables.<br />

10


Adding SCORM metadata<br />

The interface for this step has been designed according to the above mentioned approach: giving users the<br />

possibility of completing their tasks without having to learn new technologies and tools.<br />

A form directly integrated into Microsoft Word allows users to add the required metadata. The SCORM<br />

metadata (Advanced Distributed Learning, 2004a) have been classified into different groups, based on the author<br />

competency and the importance of each single data. We have identified three main roles in the process of content<br />

authoring: the lesson author (writing single units of content), the module author (collecting single units into a<br />

compound) and the editor (assembling the final LO). We have selected the relevant metadata and we have<br />

created specific interfaces for each of these roles. Moreover, we put the most important fields onto the main form<br />

and the other relevant fields in an advanced form, so as to simplify the author’s work. An example of the<br />

SCORM interface in ISA-BeL is shown in the following Figure 6.<br />

Producing the LO<br />

Figure 6. The form for Meta-Data<br />

Once the SCORM meta-data were inserted, the creative work of the author is completed and the production<br />

phase begins. In complex organizations (such as, for example, in the Project A³ staff) this phase is performed by<br />

a post-production team. Three actions have to be performed in sequence:<br />

1. The author has to save each file by using the “Save as” option of the “File” menu provided by MS Word to<br />

obtain an .htm version. This phase depends from the used word processor and it is unnecessary for some<br />

specific tools, i.e. Open Office Writer (OpenOffice, <strong>2006</strong>).<br />

2. Each of the .htm needs to be transformed into an intermediate XML format by using ISA represents<br />

fragments of raw LO. This cleaning process transforms the presentation-centered HTML automatically<br />

produced by MSWord in a valid XML file which correctly separates presentation and structure. Such a<br />

XML includes some HTML tags used with their implicit semantics to structure its content. Each file<br />

contains a portion of final LO content of the while presentational elements are completely removed. Figure<br />

7 presents the ISA on line interface allowing the user to process single files or a .zip file containing all the<br />

fragment that compose a LO.<br />

3. Finally the set of files processed by ISA are sent to the BeL system which offers two main functions: (i) to<br />

compose the LO by sequentializing the content defined by each single .xml file and (ii) to add all the<br />

elements needed for the provisioning of a Web content (i.e. presentation and navigation). Figure 8 presents<br />

the ISA on line interface that allows the user to process single files or a zip file containing all the fragments<br />

composing a LO.<br />

11


Figure 7. The ISA on line interface<br />

Figure 8. The BeL on line interface<br />

The authoring and producing phases are now completed. The LO has to be loaded onto a LCMS to be browsed.<br />

In A³ scenarios such a platform is a personalized instance of ATutor (Adaptive Technology Resource Centre,<br />

University of Toronto, <strong>2006</strong>). Figure 9 shows the page created during the process we have detailed above stepby-step<br />

as it is rendered on the Web.<br />

Accessibility evaluation<br />

Accessibility of e-learning materials which has been produced by the above mentioned tools has been widely<br />

tested by involving a group of people with disability in verifying on-line contents and services. Universality of<br />

materials has been tested by using different browser running on different platforms and specifically:<br />

MS Internet Explorer 5.0 and later (Microsoft Corp., <strong>2006</strong>),<br />

Mozilla Firefox 1.0 and later (Mozilla Project, <strong>2006</strong>),<br />

Netscape Communicator 7.0 and later (Netscape Communications Corp., <strong>2006</strong>),<br />

Lynx 2.8.4 rel. 1 (Lynx Project, <strong>2006</strong>),<br />

IBM Home Page Reader 3.0 (IBM, <strong>2006</strong>), and finally<br />

Apple Safari 1.0 (Apple Computer Inc., <strong>2006</strong>).<br />

There are two obvious considerations about such an evaluation. First of all, we have exclusively considered<br />

accessibility from a technical point of view. Pedagogical issues, including those ones on learning styles, are out<br />

of the scope of our tests. Secondly, all LOs were controlled once loaded onto the LCMS to be properly browsed.<br />

Furthermore content produced by our process is compliant to all the constraints considered by the Italian Law on<br />

Web Accessibility (Italian Parliament, 2004), meeting also WCAG 1.0 AA level (World Wide Web Consortium,<br />

1999c) and Section 508 (U.S. Rehabilitation Act Amendments of 1998, 1998) requirements.<br />

12


Figure 9. A page as it’s rendered by a LCMS<br />

Finally, accessibility of the whole process has been verified by involving in authoring and reviewing tasks blind<br />

people using the screen reader JAWS, versions 5.10 and later (Freedom Scientific, <strong>2006</strong>). The accessibility of<br />

the process itself is both a complex and a crucial objective, which creates the basis for a true inclusion of people<br />

with disabilities inside LO production teams.<br />

Conclusions and future works<br />

The need for simplicity in knowledge creation and sharing has been a basic concern in the international elearning<br />

community. This paper presents a content creation and management process allowing authors to easily<br />

produce accessible and portable LO by using personal productivity tools (e.g. a word processor).<br />

The proposed system meets the seven main goals, listed in Section 1: it permits author to simply editing<br />

contents, by starting from zero while reusing existing materials. The tool supports the best know conciliation<br />

between e-learning standard and accessibility guidelines, so as to guarantee portability through LCMS together<br />

with the respect of WCAG (AA level) and of the Italian Law on accessibility. Finally, such an high accessibility<br />

level is itself an assurance that our process produces LOs which are compliant to main Web Standards based on<br />

an uniform and simple presentation style.<br />

Compared with literature, our approach combines the idea to automatically process contents to produce SCORM<br />

compliant LOs (Advanced Distributed Learning, 2004c) with the growing need to meet accessibility<br />

requirements in e-learning (IMS Global Learning Consortium, 2002b).<br />

Main future work is devoted to extend the functionalities of the system, by offering supports in creating more<br />

complex contents to the authors (such as rich media contents and conceptual maps) by ensuring accessibility and<br />

universality of the results together with simplicity of usage.<br />

Acknowledgments<br />

This work was partially funded by MIUR (Italian Ministry of Education, University and Research) and was<br />

supported by CRIAD (www.criad.unibo.it). Authors want to thank Lorenzo Donatiello, Simone Martini, Marco<br />

Roccetti, Nelda Parisini and all the colleagues that supported this work with their precious suggestions. Finally<br />

authors want to thank ASPHI Onlus Foundation (www.asphi.it) that supported tests with users.<br />

References<br />

4system (<strong>2006</strong>). WBTExpress 5.1, retrieved June, <strong>2006</strong> from: http://www.wbtexpress.com/.<br />

13


Adaptive Technology Resource Centre, University of Toronto (<strong>2006</strong>). ATutor Learning Content Management<br />

System, retrieved June, <strong>2006</strong> from: http://www.atutor.ca/.<br />

Advanced Distributed Learning (<strong>2006</strong>). Retrieved June, <strong>2006</strong> from: http://www.adlnet.org/.<br />

Advanced Distributed Learning (2004a). Content Aggregation Model (CAM), Sharable Content Object<br />

Aggregation Model (SCORM) Version 1.3, retrieved June, <strong>2006</strong> from<br />

http://www.adlnet.org/downloads/files/67.cfm.<br />

Advanced Distributed Learning (2004b). Run Time Environment (RTE), Sharable Content Object Aggregation<br />

Model (SCORM) Version 1.3, retrieved June, <strong>2006</strong> from: http://www.adlnet.gov/downloads/files/194.cfm.<br />

Advanced Distributed Learning (2004c). Sharable Content Object Reference Model (SCORM) 2004 2nd Edition<br />

Document Suite, retrieved June, <strong>2006</strong> from: http://www.adlnet.org/downloads/70.cfm .<br />

Apple Computer, Inc. (<strong>2006</strong>). Safari Browser Home Page, retrieved June, <strong>2006</strong> from:<br />

http://www.apple.com/macosx/features/safari/.<br />

Boyle, T., Leeder, D.C., & Chase, H. (2004). To boldly GLO - Towards the next generation of learning objects.<br />

In Proceedings of the Panel session at E-Learn 2004, November 1 - 5, 2004, Washington, DC.<br />

Bradley, C., & Boyle, T. (2004). The design, development and use of multimedia learning objects. Journal of<br />

Educational Multimedia and Hypermedia, 13 (4), 371-389.<br />

Conlan, O., Dagger, D., & Wade, V. (2002). Towards a Standards-based Approach to e-Learning Personalization<br />

using Reusable Learning Objects. In Proceedings of E-Learn 2002, <strong>October</strong> 15 - 19, 2002, Montreal, Canada.<br />

Di Iorio, A., Feliziani, A. A., Mirri, S., Salomoni, P. & Vitali, F. (2005). Simply Creating Accessible Learning<br />

Object. In Proceedings of eLearning and Human-Computer Interaction: Exploring Design Synergies for more<br />

Effective Learning Experiences, INTERACT 2005 Workshop, 12-16 Sep 2005. Roma, Italy.<br />

Dolog, P., & Nejdl, W. (2003). Personalisation in Elena: How to cope with personalisation in distributed<br />

eLearning Networks. In Proceedings of International Conference on Worldwide Coherent Workforce, Satisfied<br />

Users - New Services For Scientific Information, 17-19 September 2003 Oldenburg, Germany.<br />

Dolog, P., Henze, N., Nejdl, W., & Sintek, M. (2004). Personalization in Distributed eLearning Environments. In<br />

Proceedings of WWW2004, 17-122 May 2004, New York, USA.<br />

Downes, S. (<strong>2006</strong>). E-learning 2.0 at the E-learning Forum. In Perspectives and Interactive Discussion around<br />

eLearning 2.0, Elearning Forum Meeting, February 22, <strong>2006</strong>, Menlo Park, CA.<br />

Freedom Scientific (<strong>2006</strong>). Jaws for Windows Overview, retrieved June, <strong>2006</strong> from<br />

http://www.freedomscientific.com/fs_products/software_jaws.asp.<br />

Gabrielli, S., Mirabella, V., Kimani, S., & Catarci, T. (2005). eLearning Accessibility: A Comprehensive<br />

Approach to Content Development. In Proceedings of eLearning and Human-Computer Interaction: Exploring<br />

Design Synergies for more Effective Learning Experiences, INTERACT 2005 Workshop, 12-16 Sep 2005. Roma,<br />

Italy.<br />

Horizon Wimba (<strong>2006</strong>). CourseGenie, retrieved June, <strong>2006</strong> from<br />

http://www.horizonwimba.com/products/coursegenie/.<br />

IBM (<strong>2006</strong>). IBM HPR Home Page, retrieved June, <strong>2006</strong> from http://www-<br />

3.ibm.com/able/solution_offerings/hpr.html.<br />

IEEE LTSC WG12 (<strong>2006</strong>). IEEE Learning Technology Standards Committee (LTSC), retrieved June, <strong>2006</strong> from<br />

http://ltsc.ieee.org/wg12/.<br />

IEEE LTSC WG12 (<strong>2006</strong>). Learning Object Metadata Standard Maintenance/Revision, retrieved June, <strong>2006</strong><br />

from http://ltsc.ieee.org/news/20021210-LOM.html.<br />

14


IMS Global Learning Consortium (<strong>2006</strong>). Retrieved June, <strong>2006</strong> from http://www.imsproject.org/.<br />

IMS Global Learning Consortium (2002). IMS AccessForAll Meta-data Specification, retrieve June, <strong>2006</strong> from<br />

http://www.imsglobal.org/specificationdownload.cfm.<br />

IMS Global Learning Consortium (2002). IMS Guidelines for Developing Accessible Learning Applications,<br />

retrieved June, <strong>2006</strong> from http://www.imsproject.org/accessibility/accessiblevers/index.html.<br />

Italian Parliament (2004). Law nr. 4 – 01/09/2004, Official Journal nr. 13 – 01/17/2004.<br />

Kelly, B., Phipps, L., & Howell, C. (2005). Implementing A Holistic Approach To E-Learning Accessibility. In<br />

Proceedings of the 12 th International Conference of the Association for Learning Technology, 6-8 September<br />

2005, Manchester, England.<br />

Lynx Project (<strong>2006</strong>). Lynx Home Page, retrieved June, <strong>2006</strong> from http://lynx.browser.org/.<br />

Macromedia (<strong>2006</strong>). Authorware 7, retrieved June, <strong>2006</strong> from<br />

http://www.macromedia.com/software/authorware.<br />

Microformats.org (<strong>2006</strong>). Microformats.org Home Page, retrieved June, <strong>2006</strong> from http://microformats.org.<br />

Microsoft Corp. (<strong>2006</strong>). MS Internet Explorer Home Page, retrieved June, <strong>2006</strong> from<br />

http://www.microsoft.com/windows/ie/default.mspx.<br />

Mirabella, V., Kimani, S., Gabrielli, S., & Catarci, T. (2004). Accessible e-learning material: A no-frills avenue<br />

for didactical experts. New Review of Hypermedia and Multimedia, 10 (2), 165–180.<br />

Morales, R., Leeder, D., & Boyle, T. (2005). A Case in the Design of Generative Learning Objects (GLOs):<br />

Applied Statistical Methods. In Proceedings of Hypermedia and Telecommunications (EDMEDIA), June 27-July<br />

2, 2005, Montreal, Canada.<br />

Mozilla Project (<strong>2006</strong>). Mozilla Firefox Home Page, retrieved June, <strong>2006</strong> from: http://www.mozilla.org/.<br />

Netscape Communications Corp. (<strong>2006</strong>). Netscape Browser home page, retrieved June, <strong>2006</strong> from<br />

http://browser.netscape.com/ns8/.<br />

OpenOffice (<strong>2006</strong>). Writer, retrieved June, <strong>2006</strong> from http://www.openoffice.org/product/writer.html.<br />

ReadyGo, Inc. (<strong>2006</strong>). ReadyGo Web Course Builder, retrieved June, <strong>2006</strong> from http://www.readygo.com/.<br />

Salomoni, P., & Mirri, S. (2004). A multimedia broker for ubiquitous and accessible rich media content<br />

transcoding. In Proceedings of 1st IEEE International Workshop on Net-working Issues in Multimedia<br />

Entertainment – NIME (Globecom Satellite Workshop), November 29, 2004 Dallas, TX, USA.<br />

Salomoni, P. & Mirri, S. (2005). Providing Accessible and Portable Video Lecture from Content transcoding. In<br />

Proceedings of 11th Euromedia Conference, 11-13 April 2005, Toulouse, France.<br />

Serco (<strong>2006</strong>). VirtualCampus, retrieved June, <strong>2006</strong> from http://www.teknical.com/products/virtual_campus.htm.<br />

SumTotal Systems, Inc. (<strong>2006</strong>). Toolbook, retrieved June, <strong>2006</strong> from: http://www.toolbook.com/index.php.<br />

U.S. Rehabilitation Act Amendments of 1998 (1998). Section 508, retrieved June, <strong>2006</strong> from<br />

http://www.webaim.org/standards/508/checklist.<br />

University of Bologna (2004). Ambiente Accessibile d’Apprendimento, retrieved June, <strong>2006</strong> from:<br />

https://a3.unibo.it.<br />

Vitali, F. (2003). Creating sophisticated web sites using well-known interfaces. In Proceedings of HCI<br />

International 2003 Conference, June 22-27, 2003, Crete, Greece.<br />

15


World Wide Web Consortium (1999a). Accessibility Features of SMIL, retrieved June, <strong>2006</strong> from:<br />

http://www.w3.org/TR/SMIL-access/.<br />

World Wide Web Consortium (2001a). Extensible Stylesheet Language (XSL), Version 1.0, retrieved June, <strong>2006</strong><br />

from http://www.w3.org/TR/xsl/.<br />

World Wide Web Consortium (2000). Extensible HyperText Markup Language (XHTML) Version 1.0, retrieved<br />

June, <strong>2006</strong> from http://www.w3.org/TR/xhtml1/.<br />

World Wide Web Consortium (1999b). HyperText Markup Language (HTML), Version 4.01, retrieved June,<br />

<strong>2006</strong> from http://www.w3.org/TR/html4/.<br />

World Wide Web Consortium (2001b). Synchronized Multimedia Integration Language (SMIL) 2.0, retrieved<br />

June, <strong>2006</strong> from http://www.w3.org/TR/smil20/.<br />

World Wide Web Consortium (1999c). Web Content Accessibility Guidelines 1.0. (WCAG), retrieved June,<br />

<strong>2006</strong> from http://www.w3.org/TR/WCAG10/.<br />

World Wide Web Consortium (<strong>2006</strong>). Web Content Accessibility Guidelines 2.0 W3C Working Draft 27 April<br />

<strong>2006</strong>, retrieved June, <strong>2006</strong> from http://www.w3.org/TR/WCAG20.<br />

16


Gabrielli, S., Mirabella, V., Kimani, S., & Catarci, T. (<strong>2006</strong>). A Boosting Approach to eContent Development for<br />

Learners with Special Needs. Educational Technology & Society, 9 (4), 17-26.<br />

A Boosting Approach to eContent Development for Learners with Special<br />

Needs<br />

Silvia Gabrielli, Valeria Mirabella, Stephen Kimani and Tiziana Catarci<br />

University of Rome “La Sapienza”, DIS (Dipartimento di Informatica e Sistemistica), via Salaria 113, 00198<br />

Rome, Italy<br />

Tel: +39-06-49918548<br />

Fax: +39-06-85300849<br />

gabrielli@dis.uniroma1.it<br />

mirabell@dis.uniroma1.it<br />

kimani@dis.uniroma1.it<br />

catarci@dis.uniroma1.it<br />

ABSTRACT<br />

Of late there has been a growing interest and effort toward meeting the requirements of persons with special<br />

needs. However, most of the accessibility standards and guidelines that have been proposed have been<br />

developed by adopting a domain independent and often ‘technical’ perspective. Such proposals are<br />

therefore often not sufficient to achieve accessibility goals in specific application areas such as eLearning.<br />

This paper presents a boosting approach/framework toward the development of more effective and usable<br />

accessibility indications for authors of didactic content, which are currently being fed and tested within the<br />

Italian context of the VICE project. This approach is intended to take into account the aforementioned issue<br />

and to make a step forward with respect to existing accessibility proposals and approaches in the eLearning<br />

domain. In particular, we discuss our design method for increasing the quality of eLearning materials for<br />

learners with special needs and an authoring tool, aLearning, to support eLearning content authors in their<br />

development of didactic material matching those needs.<br />

Keywords<br />

eLearning, Special needs, Accessibility, Guidelines, Didactical experts<br />

1. Introduction<br />

eLearning systems typically rely on repositories of online materials that are made available to learners and<br />

teachers for access, use or re-use. We have of late witnessed a growing commitment and effort toward providing<br />

universal access to online contents for meeting the requirements of persons with special needs, the objective<br />

being either to comply with national and international regulations on accessibility (PubbliAccesso, 2004; Section<br />

508, 1973) or to expand the number of potential users that can take advantage of online resources.<br />

However, most of standards and guidelines now available in the accessibility field (Caldwell et al., 2003; W3C,<br />

2004) have been developed by adopting a domain independent perspective, which is sometimes insufficient to<br />

achieve accessibility goals in specific application areas, such as in the case of e-learning and the creation of<br />

eContent for people with special needs. Previous studies have shown that W3C guidelines are often more suited<br />

to ensure ‘technical’ aspects of accessibility (i.e., that a visual content is readable by a blind person through the<br />

support of assistive technologies, such as screen readers) rather than ‘conceptual’ ones, which are more related to<br />

the usability and quality of experience the disabled user is provided with when accessing that specific content<br />

(Di Blas et al., 2004).<br />

In this paper we discuss a boosting approach aimed at supporting the usability and effectiveness of accessibility<br />

guidelines that eLearning authors could follow to generate appropriate materials for users with special needs (as<br />

well as to transform inaccessible contents into accessible one). This approach takes into account the issue<br />

mentioned in the foregoing paragraph and contributes to progress from existing accessibility proposals in the<br />

eLearning domain, by feeding and testing its ideas within a national project for the production and reuse of<br />

didactic material targeted at higher education and industrial contexts (VICE project, Virtual Communities for<br />

Education, CNR/MIUR Italy, http://www.progettovice.it). In particular, the contribution consists of: i) a design<br />

method/framework for increasing the quality of eLearning materials for learners with special needs, ii) an<br />

authoring tool, named aLearning, to investigate and fulfill usability requirements of eLearning content creators<br />

in their development of didactic material for learners with special needs. While acknowledging that the support<br />

accorded by the proposed authoring tool is not optimal for learners with special needs, the tool does provide a<br />

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain<br />

the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that<br />

copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work<br />

owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists,<br />

requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.<br />

17


guiding and supporting environment through which authors can realize eLearning materials of higher quality for<br />

those learners.<br />

The rest of the paper is organized as follows: Section 2 comparatively introduces our approach by relating it to<br />

previous work in the field. In Section 3 we describe the design process and the main functionalities of a tool for<br />

supporting didactic authors in the creation of accessible eLearning content. In Section 4 we present our current<br />

plan and ongoing work for evaluating the tool and methodology developed, both in terms of its usability for<br />

authors and effectiveness in reaching accessibility goals. Section 5 contains our concluding remarks.<br />

2. Related work<br />

In recent years many recommendations for the design of accessible eLearning contents have been delivered by<br />

international standard organizations, as well as private educational initiatives worldwide (W3C, 2004; ADL,<br />

2005; IMS, 2004; Hodgins & Duval, 2002). Among these we refer in particular to the efforts of NCAM, IMS<br />

Global Learning Consortium and California Community Colleges (Freed et al., 2003; IMS, 2002; CCC, 2000),<br />

for providing indications on how to create accessible learning material and raising awareness about the most<br />

appropriate approach to take when accomplishing this task. Their concern is not only that of achieving content<br />

accessibility, but they aim at showing how to address and preserve the didactical properties of eLearning<br />

resources when they are to be made accessible to learners with special needs.<br />

However, it has previously been shown that indications in the form of guidelines can be very difficult to<br />

incorporate into the everyday design or authoring practice, especially by educators who do not have prior<br />

expertise on accessibility (Dix et al., 2003; Gabrielli et al., 2004). To tackle this issue, our recent work has been<br />

addressed towards investigating effective methods and tools to make accessibility guidelines more useful and<br />

usable to eLearning authors during the creation of accessible content (Mirabella et al., 2004; Gabrielli et al.,<br />

2004). This work is based on the assumption that, compliance of a specific web content to guidelines, as it might<br />

be assessed by the most commonly used accessibility checkers, e.g., Bobby, LiFT, A-Prompt (Bobby <strong>2006</strong>;<br />

LiFT, <strong>2006</strong>; A-Prompt, <strong>2006</strong>), is not sufficient for eLearning material, since these tools mainly perform a<br />

syntactic assessment of web pages, but say very little about the adequacy of any equivalent-alternative contents<br />

created, to enable effective use of this materials by learners with special needs.<br />

The approach we have adopted is not dissimilar from other relevant proposals found in the literature that support<br />

authors in creating directly the right (i.e., accessible) version of didactic material. An example is ATutor Content<br />

Editor (ATRC, <strong>2006</strong>), part of a Learning Content Management System meant to support educators in a quick<br />

creation, reuse, packaging and distribution of web-based instructional materials. What is different in our work is<br />

the specific interest and focus on approaching the design of this type of systems from an experimental<br />

perspective. Specifically, we aim at developing an experimental prototype enabling a deeper analysis of authors<br />

usability requirements during the process of creating accessible content for learners with special needs (including<br />

different kinds of disabilities). By conducting a series of evaluation studies on the prototype use we intend to<br />

discover, and also propose solutions, to usability issues raised by the deployment of accessibility guidelines in<br />

the authoring practice. These findings will inform the current debate on how to effectively approach the design<br />

of eLearning contents when a wide range of disabilities (not only sensorial ones) is taken into account.<br />

In a previous work (Mirabella et al., 2004) we have presented a no-frills methodology to guide didactical experts<br />

during the creation of accessible eLearning content, specifically customized for learners with special needs. This<br />

methodology is based on a first categorization of the types of disabilities or impairments for the potential<br />

learners. We also characterize the types of content that are critical when it comes to making the learning material<br />

accessible. Such an undertaking would be resourceful especially to the creator of the didactical contents (e.g., a<br />

lesson), who are usually not well acquainted with the issues or problems of accessibility. We then associate the<br />

realized categories of disabilities with the various types of critical content. At the point where a particular<br />

disability intersects with a particular critical content, we analyse the range of accessibility barriers and<br />

opportunities presented. In the process, we are better positioned to develop indications that could be resourceful<br />

especially to the creator of the didactical contents of the learning material. It is worth noting that such indications<br />

can guide in specifying alternative content that is characterized by matching the following requirements of being<br />

useful, appropriate and effective, where:<br />

Usefulness is closely related to the concept of necessity of the content. The objective is for an author to<br />

eliminate the contents that are not necessary, in order to increase the overall usability of the didactic<br />

modules, especially in the case in which alternative content types (or formats) have also been introduced.<br />

18


Appropriateness is related to the selection of the right contents while taking into account the characteristics<br />

and requirements of a certain type of disability.<br />

Effectiveness is related to the capacity of the didactic content to enabling the achievement of the learning<br />

goals.<br />

In the rest of this section, we first discuss how we categorized the types of disabilities or impairments and the<br />

critical content types considered so far, as well as the relations between them. We then exemplify some possible<br />

applications of our proposed guidelines and no-frills methodology.<br />

2.1 The disabilities, the contents and their critical relations<br />

With critical learning resource types we refer to typical didactical contents (e.g., of an eLearning module) that<br />

can affect accessibility. For example a graph (but also a diagram, image or table) is a critical didactical content<br />

for people who present cognitive disabilities (such as dyslexia) because of problems with ordering and cluster<br />

identification, thus it is not accessible as it is. Several lists of didactical contents are reported by some of the<br />

most notable organizations involved in eLearning standardization and specifications.<br />

The IEEE Learning Object Metadata (LOM) (LTSC, 2004) is one of the most comprehensive schemes<br />

developed for the description of Learning Objects. In the Educational section of the LOM, there is a list of<br />

Learning Resource Types including: exercise, simulation, questionnaire, diagram, figure, graph, index, slide,<br />

table, narrative text, exam, experiment, problem statement, self-assessment, and lecture.<br />

We have partially adopted this list, in agreement with the opinion that "LOM allows this (the Learning Resource<br />

Types) element to be understood as designating types or formats of content, as well as potential uses of this<br />

content. Also, the values recommended by the LOM exclude many important types of content, and also exclude<br />

many educational applications of content" (CanCore, <strong>2006</strong>). In our proposal we start with an investigation of<br />

only the elements from the LOM list that are both critical from an accessibility perspective and that are format<br />

independent.<br />

We consider the following elements from the LOM proposal to be format dependent: exercise, simulation,<br />

questionnaire, exam, experiment, problem statement and self-assessment. Their format, in fact, can vary<br />

according to the different types of contents (images, text, tables, etc.) they may include. Moreover, we consider a<br />

slide and a lecture to be non-digital content with respect to the LOM definition. So far, our approach translates<br />

visual content to text and therefore, here, we do not list text and index as part of critical content.<br />

Consequently, we focus on four types of critical didactical content from the LOM model: diagram, figure, graph,<br />

and table. We also consider another set of critical learning resource types which is derived from a list provided<br />

by the CPB/WGBH National Center for Accessible Media (NCAM). In (Freed et al., 2003), CPB/WGBH<br />

presents eight guidelines in relation with the same number of learning content types: images, multimedia, forms,<br />

tables, text-books, interactivity, graphs and math.<br />

From the CPB/WGBH NCAM we investigate: multimedia and math/scientific expressions. We do not take into<br />

account forms because the kind of accessibility issues they present are more related with the format rather than<br />

with the semantic meaning of didactical content. We also exclude textbooks, since we assume they are in textual<br />

form, and interactivity, since it is not a kind of content. We are aware that multimedia contents present complex<br />

technical and pedagogical issues to be addressed for ensuring their accessibility. Initially, we have considered<br />

different media as aggregated in a same category (multimedia) to refer to their semantic meaning more than to<br />

their particular format. Our aim is to provide clear indications, for example, on how to create effective captions<br />

and audio descriptions for video contents typically employed as eLearning materials. More detailed analysis of<br />

different components of this category will be considered in our future work in compliance with related efforts in<br />

the field e.g., Rich Media Accessibility (NCAM, <strong>2006</strong>).<br />

The six elements selected above are then mapped and analysed in relation to the following main types of<br />

disabilities for potential learners: visual, hearing, motor, cognitive-language impairments. It is worth observing<br />

within each of these categories there are many variations and degrees of impairment that require specific<br />

strategies to be addressed. Also, there could be learners that exhibit more than one type of disability, like senior<br />

citizens having both sensorial and cognitive impairments. However, our current efforts are oriented toward<br />

providing didactic experts with a wide spectrum of indications supporting the removal of accessibility barriers<br />

from teaching materials for the main categories of disabilities that might be present in our target user population.<br />

19


As an example, to improve the accessibility of a figure for learners with cognitive disabilities, we indicate how<br />

to provide significant alternative content that can properly explain that figure, possibly including a description of<br />

the overall meaning and contribute provided by that figure within the teaching material in which it is embedded.<br />

For the needs of learners with motor impairments, we raise awareness and provide support to authors on how to<br />

make operations on mathematical and scientific expressions more easily accessible, since this type of learners<br />

may have difficulties in using input modalities, such as the mouse, to insert and modify symbols or strings, as<br />

well as to keep track of the content already inserted.<br />

Table 1 which is adapted from (Mirabella et al., 2004) reports schematically the most relevant mappings<br />

identified between disabilities and type of content considered so far, which are also of particular relevance to the<br />

didactic materials currently dealt with in the VICE project.<br />

Table 1: Relevant mappings between disability types and critical contents<br />

Type of Disabilities<br />

Type of Content Visual Hearing Motor Language/Cognitive<br />

Diagram • •<br />

Figure • •<br />

Graph • •<br />

Table • •<br />

Multimedia • • • •<br />

Math/scientific expression • • • •<br />

Figure 1: Example of Guideline for type of Content Graph w.r.t. Visual Disabilities<br />

20


2.2 The guidelines and their application through the no-frills methodology<br />

Guidelines were developed for each relevant mappings identified between critical contents and types of<br />

disabilities, as discussed in the previous section. They included detailed indications on how to create equivalentalternative<br />

versions of inaccessible contents, as well as hands-on examples for authors on how to proceed during<br />

the repairing (Gabrielli et al., 2004; Mirabella et al., 2004). Fig. 1 below reports a selection of indications<br />

contained in the guideline developed for the type of content Graph w.r.t. Visual disabilities.<br />

In practice, the guidelines are expected to support authors as they apply the no-frills methodology (Mirabella et<br />

al., 2004) in both cases in which: a) they are creating new didactic materials to be added to a content repository,<br />

b) to reuse already existing materials and transform them into accessible ones for learners with special needs. In<br />

particular, the no-frills methodology provides an avenue for guiding didactical experts in making the most of<br />

their didactical experience when choosing the necessary and alternative didactical content that can fit the<br />

requirements of disable learners. A brief description of the different actions an author should take while applying<br />

the no-frill methodology is provided below.<br />

1. Initially, the method requires the categorization of the potential learners, in terms of disabilities or<br />

impairments for the specific didactical module considered.<br />

2. The next step involves identifying the content types the didactical module is made of. For every type of<br />

content, the methodology analyses its impact on each of the categories of the learners in terms of physical<br />

accessibility (the learner can practically access the content) and logical accessibility (the learner can<br />

effectively access the content). In the first case, the focus is on accessibility as intended in a rather rigid<br />

sense, whereas in the second case, the focus is on accessibility related more to didactic effectiveness.<br />

3. The third step involves exploiting opportunities for inserting alternative content that corresponds to the<br />

critical content under consideration for some category of learners. In particular, we consider that the didactic<br />

content may be considered as:<br />

a) optional, in which case the content is not essential to the realization of the module’s objective;<br />

b) mandatory, in which case the content is essential or relevant to the realization of the module’s<br />

objective. In the mandatory case, the didactic expert may also specify whether the level of<br />

accessibility is acceptable toward realizing the module’s objective. If the accessibility level is not<br />

considered acceptable for accomplishing the objective, the didactic expert may decide whether to<br />

translate the content or substitute it with some other relevant alternative content better able to<br />

contribute to the realization of the same objective.<br />

2.3 Main issues raised in the authoring practice<br />

The guidelines and methodology proposed were initially tested with a small group of eLearning authors during a<br />

formative evaluation study that we conducted to assess the principle on which our approach is based (Gabrielli et<br />

al., 2004). The evaluation scenario involved five didactic experts who were asked to work individually at<br />

inspecting the contents of a Learning Object of the type used in the VICE repository, which consisted of 4 basic<br />

modules of material on computer science topics (European Computer Driving License material). These included<br />

a series both text and the collection of critical contents of the types mentioned above: figures, diagrams, graphs,<br />

tables, multimedia, and math/scientific expressions.<br />

The main task didactical experts were asked to perform was to examine and to identify any content that might<br />

have been critical for disable learners (according to the categories reported in Table 1) and to make it accessible<br />

to them by following the accessibility guidelines and no-frills approach described above. These were made<br />

available in the authoring environment as indications contained within HTML page whose links inspectors could<br />

activate whenever required. Throughout the study, suggestions for alternative contents required were reported on<br />

paper protocols by the didactic experts, then transformed into digital format and added to the LO by the study<br />

moderator, so that subsequent participants had the opportunity of evaluating the appropriateness of the<br />

alternative content previously created by other authors. Didactic experts were asked to think aloud during their<br />

task performance (lasting approximately 30-45 minutes) and briefly interviewed at the end of the task to collect<br />

their impressions about the guidelines and methodology used, as well as about the difficulties raised during their<br />

application. As a result we observed that didactic experts found quite difficult and time demanding the<br />

translation or substitution of critical content with appropriate alternatives and accessible versions of it, especially<br />

when it involved the use of diagrams, graphs, tables and scientific-math expressions. Typically, experts used to<br />

develop and apply a limited but consistent set of ‘translation’ strategies throughout the whole duration of their<br />

activity (e.g., inserting textual description for visual contents), whilst elaboration of more original alternatives<br />

was rarely found (although indications in the guidelines were specifically prompting them to undertake this<br />

21


option). Although this finding might be related to experts lack of familiarity with accessibility issues, it also<br />

highlighted the need for content creators to be better supported and trained, for instance by interface<br />

functionalities, in the acquisition of a specific expertise on accessibility. However, we derived from observations<br />

collected in this study that any accessibility support provided during such a task should not constrain authors’<br />

activity in a very prescriptive way, but enable flexible and creative use of the indications made available.<br />

To achieve the design of this kind of support, we decided to develop a user-friendly interface for eLearning<br />

authors. This serves to present accessibility indications in a more interactive way and to enable authors a more<br />

intuitive and easier application of our no-frills methodology during the creation of accessible contents.<br />

3. aLearning: Usable Presentation of Accessibility Guidelines<br />

The experimental prototype we have designed is an accessibility interface we named aLearning. The main<br />

objective of the tool is to enable exploration of more usable presentation of accessibility guidelines in the context<br />

of eLearning content creation. The tool (developed in Java) is integrated into an application server which is a<br />

key component of the VICE architecture and provides also other functionalities, such as a metadata modeler and<br />

editor (Fig. 2). The application server is responsible of ensuring that the eLearning contents retrieved from the<br />

VICE repository are packetized into SCORM compliant LOs, to be then uploaded on Learning Management<br />

Systems, such as Atutor (ATRC, <strong>2006</strong>) for instance, and presented as web-based courses.<br />

The support provided by aLearning is also available in the case where authors decide to retrieve materials not<br />

originally created for users with special needs. This is not the approach generally recommended by the<br />

accessibility literature for generating high quality didactic contents, however it could bring some important<br />

advantages in specific eLearning contexts, such as costs and time savings in the production of the required<br />

materials. Initial investigations conducted during the user requirements analysis of the VICE project also<br />

reported authors increased willingness to work at transforming contents into accessibility ones in return of the<br />

possibility of reusing existing materials for the design of university courses.<br />

Figure 2: aLearning and the VICE architecture<br />

A user-centred design approach was adopted to make aLearning interface intuitive to use even for authors<br />

without a specific expertise on accessibility and/or without a technical background. A main characteristic of the<br />

22


tool is to facilitate the creation of accessible and usable eLearning contents by making automatic or semiautomatic<br />

the application of any technical step of our no-frills methodology that would not benefit from human<br />

intervention or expertise, such as the identification of critical contents within the eLearning material. However,<br />

the tool would prompt, inform and capitalize on authors’ didactic knowledge and decision making when<br />

addressing the non-technical steps of the method (e.g., creating an alternative-equivalent version of the content).<br />

The design process went through a task analysis phase and a series of low-mid tech prototyping activities to<br />

develop the interface features required, that we briefly describe in the following.<br />

3.1 Overview of aLearning Prototype<br />

aLearning interface mainly supports authors by:<br />

Automatically identifying and marking any critical and inaccessible content within the eLearning material.<br />

For example, when a graphic file is detected (e.g., by detecting the file extension) the tool first asks the<br />

author to classify it as a diagram, figure, graph etc., then it presents all relevant guidelines to ensure<br />

accessibility of that content according to the category assigned by the author. In this way the authoring or<br />

repairing process performed on eLearning material is led by the consideration of didactically relevant<br />

categories of contents (such as diagrams, figures, graphs, ...), instead of the web related ones (like framesets,<br />

scripts, links, ...) typically used by common tools for accessibility checking and repairing (Bobby, <strong>2006</strong>;<br />

LiFT, <strong>2006</strong>; A-Prompt, <strong>2006</strong>; NCAM, <strong>2006</strong>).<br />

Enabling the author to select and start repairing a critical content (e.g., a graph) by choosing among three<br />

alternative modalities: i) directly clicking on the critical content as it is marked within the didactic module’s<br />

pages that are displayed, ii) selecting that content from a category list of learning resources reported on a<br />

frame window, iii) starting a step-by-step repairing procedure for the whole sequence of inaccessible<br />

contents identified. These different modalities are expected to provide authors with more flexibility on how<br />

to complete the repairing process (also in terms of its timescale), differently from the file-by-file repairing<br />

process typically supported by other tools.<br />

Explaining to the user why a specific content is inaccessible, prompting the user to classify it as optional or<br />

mandatory for the objectives of the course (according to our no-frills methodology), supporting different<br />

ways of creating alternative versions of the content if it is mandatory, providing links to more detailed<br />

information (guidelines and examples) on how to create an appropriate alternative representation of it.<br />

Fig. 3 below presents an example of how an author would be guided to check and repair the critical content<br />

Graph according to the guideline reported in section 2.<br />

Figure 3: Example of aLearning support to authors for repairing the type of content Graph<br />

Beyond these basic functionalities of the tool, we are also investigating how aLearning interface could support<br />

authors in the generation of appropriate metadata for the accessible contents created. By pursuing compliance<br />

with existing standards and specifications (IMS AccessForAll, http://www.imsglobal.org/accessibility), as well<br />

as by acknowledging current open issues in the field (e.g., adaptability of content), we are analysing the<br />

opportunity for a possible extension of LOM categories that would specifically address accessibility properties<br />

23


of eLearning content and enhance the retrieval of accessible resources from the VICE LOs repository. Moreover,<br />

according to our design approach, any interface support for the editing of metadata should include clear and<br />

detailed explanations of the metadata fields required, to make it easier for authors without previous expertise on<br />

standards technicalities or vocabulary, a straightforward understanding of the metadata information to be<br />

included.<br />

Our current efforts are addressed toward refining and improving the usability and learnability of the tool<br />

developed so far, to speed up not only any authoring and repairing process performed by the user, but also<br />

authors’ acquisition of expertise on accessibility by means of navigation through aLearning functionalities.<br />

4. Evaluation Plan and Work in Progress<br />

As we mentioned in Section 2, we have already conducted some formative evaluation activities to assess the<br />

soundness of our approach and guidelines as a support to the eLearning authoring practice. However, we plan<br />

now to conduct a more thorough evaluation on how effective the method and aLearning tool developed are in<br />

supporting the creation of accessible contents for learners with special needs. This can be achieved by<br />

performing comparative studies with other existing accessibility repairing tools and evaluation strategies.<br />

Current efforts toward a detailed verification of web content accessibility suggest recur to a combination of<br />

techniques and tools, such as:<br />

the use of more than one automatic check-repairing tool,<br />

a manual verification of content compliance with accessibility guidelines (usually performed by accessibility<br />

experts),<br />

some testing phases directly involving users with special needs or accessibility experts in using the content<br />

created (Lang, 2003; Mankoff et al., 2005; Bertini et al., 2005).<br />

The main motivation for adopting this comprehensive strategy to accessibility verification is that the different<br />

methods mentioned above provide useful information as well as possible limitations (e.g., in terms of resources<br />

to be invested for the evaluation), thus their combination is often likely to produce better results at a lower cost.<br />

Inspired by these considerations we are currently conducting the following activities to validate the quality of the<br />

proposed approach to pursue accessibility in the eLearning domain:<br />

A comparative assessment of aLearning features with those provided by other well-known tools for the<br />

authoring of accessible eLearning material, e.g., ATutor Content Editor (ATRC, <strong>2006</strong>), or accessibility<br />

check and repairing, e.g., A-Prompt and LiFT (A-Prompt, <strong>2006</strong>; LiFT, <strong>2006</strong>), in terms of their usability and<br />

support provided to users. In the evaluation studies we observe authors in the activity of transforming<br />

eContent (e.g., LOs resources retrieved from the VICE repository) into accessible one, by means of the<br />

authoring and check-repairing tools mentioned above. Application of usability criteria (such as<br />

effectiveness, efficiency and user satisfaction) steer the assessment process.<br />

The involvement of a number of accessibility experts in manually checking if a correct application of the<br />

eLearning guidelines and no-frills methodology has been performed by authors in phase 1. This evaluation<br />

is inspired to methodological studies typically carried out in the usability field to assess the validity of<br />

inspection techniques, such as Heuristic Evaluation and Cognitive Walkthrough (Cockton et al., 2003). In<br />

our case, the aim is mainly to verify which possible improvements to our guidelines are required to ease<br />

their understandability and deployment.<br />

The direct involvement of learners with special needs in accessing the educational materials produced in<br />

phase 1 (by using the assistive technologies they are most familiar with) to verify the quality of the<br />

eLearning contents generated by authors, with special attention to the equivalent-alternative versions of<br />

didactic material they have created.<br />

Observations collected from these evaluation studies will enable us to feed any future refinement of our<br />

accessibility approach, as well as to further inform the iterative design of aLearning.<br />

5. Conclusion<br />

In this paper, we have highlighted some main limitations of the existing accessibility guidelines for eLearning.<br />

The methodological approach we have proposed contributes to improve usability and effectiveness of<br />

accessibility indications provided to eLearning authors. This kind of support is particularly important to deliver<br />

in contexts such as our VICE project (which is targeted at higher education and professional training<br />

environments) where there is also the need to comply with more restrictive accessibility regulations<br />

24


(PubbliAccesso, 2004) than the ones that have inspired the design of other existing accessibility tools, e.g.,<br />

ATutor in Canada (ATRC, <strong>2006</strong>). Specifically, in this paper we have presented motivations and current efforts<br />

towards designing more usable and interactive tools for the presentation of accessibility guidelines as supports to<br />

the authoring practice. One of the main objectives of the approach adopted so far has been to encourage the<br />

acquisition of expertise on accessibility by creators of eLearning resources, no matter what their professional<br />

expertise or previous technical abilities are. Another main focus has been to promote, through a correct<br />

application of the methodology presented, that the eLearning contents and experience eventually delivered to<br />

learners with special needs reach the same levels of quality and effectiveness of the ones provided to nondisabled<br />

learners. Our current work entails a validation of the approach proposed, by which we expect to inform<br />

its future refinement as well as its possible extension to a larger set of disabilities and types of contents.<br />

Acknowledgments<br />

This work has been supported by the VICE project (Virtual Communities for Education, CNR/MIUR, Italy,<br />

http://www.progettovice.it/). We would like to thank Loredana De Giovanni for her comments and support<br />

provided during the preparation of this work.<br />

References<br />

ADL (2005). Advanced Distributed Learning, retrieved June 29, <strong>2006</strong> from http://www.adlnet.org/.<br />

A-Prompt (<strong>2006</strong>). University of Toronto, A-Prompt, retrieved March 14, <strong>2006</strong> from<br />

http://aprompt.snow.utoronto.ca/.<br />

ATRC (<strong>2006</strong>). ATutor, retrieved June 29, <strong>2006</strong> from http://www.atutor.ca/atutor/index.php.<br />

Bertini, E., Billi, M., Burzagli, L., Gabbanini, F., Gabrielli, S., Graziani, P., & Palchetti, E. (2005). Testing<br />

Accessibility in Mobile Computing. International Conference on Human-Computer Interaction 2005, July 2005,<br />

Las Vegas, USA.<br />

Bobby (<strong>2006</strong>). Watchfire Corporation, Bobby, retrieved March 14, <strong>2006</strong> from http://bobby.watchfire.com/.<br />

Caldwell, B., Chisholm, W., White, J., & Vanderheim, G. (2005). Web Content Accessibility Guidelines 2.0,<br />

Working Draft 23 November 2005, retrieved June 29, <strong>2006</strong> from http://www.w3.org/TR/WCAG20/.<br />

CanCore (<strong>2006</strong>). CanCore Guidelines Version 1.9: Educational Category, retrieved March 14, <strong>2006</strong> from<br />

http://www.cancore.ca/guidelines/1.9/CanCore.<br />

CCC (2000). Guidelines for Producing Instructional and Other Printed Materials in Alternate Media for Persons<br />

with Disabilities, California Community Colleges. retrieved June 29, <strong>2006</strong> from<br />

http://www.htctu.net/publications/guidelines/altmedia/altmedia.htm<br />

Cockton, G., Lavery, D., & Woolrych, A. (2003). Inspection-based Evaluations. In Jacko, J. A. & Sears, A.<br />

(Eds.), The HCI Handbook, Lawrence Erlbaum Assoc., 1118-1138.<br />

Di Blas, N., Paolini, P., & Speroni, M. (2004). “Usable Accessibility” to the Web for Blind Users. In Adjunct<br />

Proceedings of the 8 th ERCIM Workshop on User Interfaces for All, June 2004, Vienna, Austria,<br />

http://ui4all.ics.forth.gr/workshop2004/publications/adjunct-proceedings.html.<br />

Dix, A., Finlay, J., Abowd, G., & Beale, R. (2003). Human-Computer Interaction, Hillsdale, NJ: Prentice Hall.<br />

Freed, G., Rothberg, M., & Wlodkowski, T. (2003). Making Educational Software and Web Sites Accessible.<br />

Design Guidelines Including Math and Science Solutions, retrieved March 14, <strong>2006</strong> from<br />

http://ncam.wgbh.org/cdrom/guideline.<br />

Gabrielli, S., Mirabella, V., Kimani, S., & Catarci, T. (2004). Steering the Development of Accessible e-<br />

Learning Content. In Proceedings of the 3 rd ECEL Conference, Paris, France, 517-526.<br />

25


Hodgings, W. & Duval, E. (2002). IEEE LTSC Learning Object Meta-data LOM 1484.12.1 Vl Final Draft,<br />

retrieved March 14, <strong>2006</strong> from http://ltsc.ieee.org/wg12/files/.<br />

IMS (2002). IMS Guidelines for Developing Accessible Learning Applications, Version 1.0, White Paper,<br />

retrieved March 14, <strong>2006</strong> from http://www.imsproject.org/accessibility/.<br />

Lang, T. (2003). Comparing website accessibility evaluation methods and learnings from usability evaluation<br />

methods, retrieved March 14, <strong>2006</strong> from http://www.peakusability.com.au/pdf/website_accessibility.pdf.<br />

LiFT (<strong>2006</strong>). UsableNet, LiFT, retrieved March 14, <strong>2006</strong> from http://www.usablenet.com/.<br />

LTSC (2004). IEEE LTSC Learning Object Meta-data LOM_1484_12_1_v1_Final_Draft, November 2004,<br />

retrieved March 14, <strong>2006</strong> from http://ltsc.ieee.org/wg12/files/.<br />

MAGpie (<strong>2006</strong>). NCAM, MAGpie, retrieved March 14, <strong>2006</strong> from http://ncam.wgbh.org/webaccess/magpie/.<br />

Mahemoff, M. J., & Johnston, L. J. (1998). Principles for a Usability-Oriented Pattern Language. In Proc. of<br />

Australian Computer Human Interaction Conference OZCHI’98, Adelaide: IEEE Computer Societey, Los<br />

Alamitos, 132–139.<br />

Mankoff, J, Fait, H., & Tran, T. (2005). Is Your Web Page Accessible? A Comparative Study of Methods for<br />

Assessing Web Page Accessibility for the Blind. Proc. of CHI 2005, April 2-7, 2005, Portland, USA, 41-50.<br />

Mirabella, V., Kimani, S., Gabrielli, S., & Catarci, T. (2004). Accessible e-Learning Material: A No-Frills<br />

Avenue for Didactical Experts. The New Review of Hypermedia and Multimedia, 10 (2), 1-16.<br />

NCAM (<strong>2006</strong>). Rich Media Accessibility, retrieved June 29, <strong>2006</strong> from<br />

http://ncam.wgbh.org/richmedia/tutorials.<br />

PubbliAccesso (2004). PubbliAccesso, Italian Law 9 January 2004, n. 4, retrieved March 14, <strong>2006</strong> from<br />

http://www.pubbliaccesso.it/normative/legge_20040109_n4.htm.<br />

Section 508 (1973). Section 508 of the Rehabilitation Act, amended 1998, retrieved March 14, <strong>2006</strong> from<br />

http://www.section508.gov.<br />

Van Welie, M., Van der Veer, G. C., & Eliëns, A. (2000). Patterns as Tools for User Interface Design.<br />

International Workshop on Tools for Working with Guidelines, 7-8 <strong>October</strong> 2000, Biarritz, France, 313-324.<br />

W3C (2004). Web Content Accessibility Guidelines 2.0, W3C Recommendations, Working Draft 29, November<br />

2004, retrieved March 14, <strong>2006</strong> from http://www.w3.org/TR/WCAG20/#overview-design-principles.<br />

26


De Carolis, B., Pizzutilo, S., Cozzolongo, G., Drozda, P., & Muci, F. (<strong>2006</strong>). Supporting Students with a Personal Advisor.<br />

Educational Technology & Society, 9 (4), 27-41.<br />

Supporting Students with a Personal Advisor<br />

Berardina De Carolis<br />

Researcher ,Dipartimento di Informatica – Università di Bari, Via Orabona 4, 70126 Bari, Italy<br />

decarolis@di.uniba.it<br />

Sebastiano Pizzutilo<br />

Associated Professor, Dipartimento di Informatica – Università di Bari, Via Orabona 4, 70126 Bari, Italy<br />

pizzutilo@di.uniba.it<br />

Giovanni Cozzolongo, Pawel Drozda and Francesca Muci<br />

Phd Student, Dipartimento di Informatica – Università di Bari, Via Orabona 4, 70126 Bari, Italy<br />

cozzolongo@di.uniba.it<br />

pawel@di.uniba.it<br />

muci@di.uniba.it<br />

ABSTRACT<br />

Recently, many national and international initiatives show an increasing interest in promoting distance<br />

education interventions by Universities. These directives aim at supporting students during their studies<br />

providing them with personalized solutions to their problems related not only to the fruition of on-line<br />

courses but also to orientation issues. In this paper we present a general architecture of an Embodied<br />

Conversational Agent (ECA) that has the main aim to assist students by providing personalized suggestions.<br />

This ECA represents the student interface in interacting with a Virtual University environment able to<br />

provide different types of services. In particular to support mobility, the ECA architecture has been<br />

conceived in order to run also on the student handheld device. The paper discusses the design and technical<br />

issues involved in developing this personal agent and the results of an evaluation study aiming at assessing<br />

the impact of conversational agents on the effectiveness of the interaction.<br />

Keywords<br />

Personal Advisor, Embodied Conversational Agents, M-learning, Multiagent-Systems, Virtual Universities<br />

Introduction<br />

In this paper we present a general architecture of an Embodied Conversational Agent (ECA) that has the main<br />

aim to assist students by providing personalized suggestions. These suggestions can be related not only to the<br />

fruition of on-line courses but also to orientation issues regarding student daily life.<br />

The motivations at the basis of this work come from the directive of the Italian Universities that aims at<br />

promoting interventions for improving mentoring and orientation services. This directive aims at following<br />

students during the entire course of study, giving them personalized suggestions about different topics and<br />

helping them to take decisions about their formative process. In particular, taking as a reference the role of an<br />

advisor in our Department, his/her main goal consists in assisting and orienting the students focusing on:<br />

suggestions about orientation choices, personalization of the study curricula, removal of “obstacles” during the<br />

course of study, identification of appropriate and interesting research fields for the thesis, suggestions on how to<br />

make students more participative in their formative process. At the first year, each student is assigned to an<br />

advisor, chosen between available professors by the Department Council. Each advisor follows a group of<br />

students during the entire course of study. After a trial period, we noticed that this service was not fully exploited<br />

by students and we investigated the reasons related to this phenomenon. In order to assess the motivations, we<br />

made a user study consisting in a questionnaire (150 subjects in total) aiming at understanding which was the<br />

user’s expectations about the advisor role and at assessing why students were not using this orientation service.<br />

Analyzing the result of this survey, it came out that the main reason was related to the difficulty to find the<br />

advisor always available when needed (professors teach, do research, are involved in meeting, etc.) and also to<br />

the shyness of some student that are afraid of asking questions about their problems to professors.<br />

Taking these motivation and the ubiquitous computing vision (Weiser, 1991) into account, we decided to<br />

develop VU-MAS, a Virtual University Multi-Agent System (MAS) with the aim of supporting students during<br />

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain<br />

the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that<br />

copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work<br />

owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists,<br />

requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.<br />

27


the entire course of study. This system is able to support students by providing a interface to access to several<br />

services by combining knowledge stored in distributed sources whenever needed.<br />

Each student can interact with VU-MAS using a personal agent, called MyCoach, represented as an ECA. The<br />

main goal of this agent is to monitor the student activities, following his/her learning improvements, but also to<br />

select useful material according to the recognized student’s goals and needs. The agent is also capable to<br />

proactively provide the student with useful suggestions whenever it is needed. As it is designed to run on a<br />

smart phone or a PDA, this agent combines e-learning capabilities with mobile computing, thus realizing an mlearning<br />

experience where the student can feel always in touch with his advisor.<br />

The paper is structured as follows: after a brief overview on the related work, we will briefly illustrate the VU-<br />

MAS architecture, focusing on the description of the interaction aspects of the system. Then we will describe<br />

how we extended the Mind-Body architecture, developed in the context of the MagiCster project (de Rosis et al.,<br />

2003), in order to implement MyCoach. In order to assess the impact of the conversational agent on the<br />

effectiveness of the interaction we performed an evaluation study whose results are reported in the related<br />

section. The paper ends with some concluding remarks and some pointers to future work.<br />

Related Work<br />

Developing more human-like systems seems to improve interaction by establishing a more engaging relation<br />

with the agent (Dehn and van Mulken, 2000). In this role, ECAs offer to people the possibility to relate with<br />

computer media at a social level (Reeves and Nass, 1996) and, therefore, they have potential as facilitators for<br />

this type of interventions thus making the interaction more effective and enjoyable (Moreno et al., 2001).<br />

According to research and evaluation studies in the field of intelligent interfaces, ECAs (Gratch et al.,2002) have<br />

shown to be a good interaction metaphor when acting in the role of counselors (Marsella, Johnson, & LaBore<br />

2003), personal trainers (Bickmore, 2003), or healthy living advisors (de Rosis et al., 2003). Indeed, ECAs have<br />

the potential to involve users in a human-like conversation using verbal and non-verbal signals for providing<br />

feedback, showing empathy and emotions in their behavior (Cassell, 2001). Due to these features, an ECA can<br />

be successfully employed as interaction metaphor in the pedagogical domain (Johnson, LaBore & Chiu, 2004)<br />

and in other domains where is important to settle long term relation with the user (Bickmore & Picard, 2005).<br />

In particular, as far as the interaction with a computational environment supporting the learning process is<br />

concerned, the use of conversational agents can be envisaged in the following roles (Roda, Angehrn and Nabeth,<br />

2001): i) as advanced helping facilities ; ii) as personal coaches equipped with specific domain knowledge; iii) as<br />

role-playing actors in simulated environments (i.e. 3D virtual worlds). When an ECA acts as a personal coach it<br />

has to show a personality appropriate to its role, the ability to guide the user, to reason about user actions, to<br />

provide useful suggestions and to communicate through a realistic, interactive interface.<br />

Moreover, even if ECAs have shown to have a good impact on settling an emphatic relation with the user (de<br />

Rosis et al., 2005; Cassell et al. 2000), involving them in a deeper and intimate interaction, it is difficult to<br />

communicate with these agents whenever needed (i.e. when the user is not in front of a computer but he/she has<br />

the need to get suggestions and advices). For example, DESIA (Johnson, LaBore and Chiu, 2004), is a step in<br />

this direction. This agent, presented in Carmen’s Bright IDEAS, has been adapted for running successfully on a<br />

handheld device in order to assist in a psychosocial intervention aimed at providing training in problem solving<br />

skills (Marsella, Johnson & LaBore, 2003). Therefore, if we want to support student in a continuous way the<br />

personal advisor should be available and accessible also on a mobile device. Our work represents a step in this<br />

direction.<br />

VU-MAS Architecture Overview<br />

A Virtual University should be able to support students from different points of view, during their formative<br />

process. It has to help students not only with online courses and material, but also with logistic information and<br />

tuition support. (Cunha, Tavares & Ferreira, 2005). Following this vision, the Virtual University MAS is<br />

composed by agents that provide information services of different kind.<br />

28


Student-<br />

Profiles<br />

MyC-1<br />

Requests /<br />

Profile<br />

Student-<br />

Profiles<br />

Strategies<br />

/resources<br />

MyC-n<br />

VA-1<br />

Strategies<br />

/resources<br />

Response /<br />

APML /<br />

Mp3<br />

VA-m<br />

Figure 1. VU-MAS architecture<br />

SOAP /<br />

HTTP<br />

VA-m = Virtual Advisor<br />

SA-k = Service Agent<br />

MyC–n = MyCoach Agent<br />

SA-1<br />

SA-k<br />

The agents in VU-MAS are grouped into three main categories: MyCoach, Virtual Advisor and Service Agents.<br />

MyCoaches (MyC-s) are personal agents that manage the student profile and the interaction between the student<br />

and his advisor, as we will show in the next section. In VU-Mas there will be a personal agent for each student.<br />

Virtual Advisors (VA-s) have the role of their real world counterpart. In fact, they represent the teachers when<br />

they are in the role of advisors. They can access a knowledge base (KB) and use a reasoning engine to decide<br />

what to suggest. The KB contains information about the students and the strategy used to advise according to<br />

student features. VA-s can store every activity of the students, when they are connected to VU-MAS, either<br />

using MyC agents or a web applications, updating the students’ profiles accordingly.<br />

Service Agents (SAs) are specialized in providing a specific service, such as administration office, a library or<br />

even a teacher, and they can be either simple request/response agents, or more complex ones.<br />

Every human advisor can access and update the KB using his/her VA. He/she is responsible for a set of students.<br />

The students use their personal device to connect to VU-MAS. This is a FIPA compliant MAS, in which the<br />

communication between MyC and VA agents is made using ACL messages (FIPA,<strong>2006</strong>). The SAs can be either<br />

agents or webservices (Greenwood, Buhler & Reitbauer, 2005). They answer to requests using standard<br />

protocols, such as SOAP and HTTP (W3C, 2004).<br />

In VU-MAS, each student is associated to one advisor. The first time the student has to download his MyC agent<br />

on his personal device, from the VU-MAS web site. Starting from that moment a Virtual Advisor agent is<br />

associated to the student who will be able to interact using MyC ECA and, according to the student year of<br />

course, passed exams or specialization level, a stereotype is assigned to him/her. This stereotype is used to set<br />

the initial student background knowledge and interests. After the agent initialization, the body of the user<br />

stereotype is copied in the student profile, which is updated according to the student interaction with VU-MAS.<br />

The profile is used for adapting the information to the student situation and recording the interaction history with<br />

the system.<br />

The profile of the students contains personal data (name, birthday, etc.), information about the student<br />

curriculum (courses, exams, marks, etc.), preferences, hobbies and interests. Each MyC agent running on the<br />

personal device manages and stores the profile of the student in a xml structures, formalized using a common<br />

representation language. We decided to use the situational statement language UserML from UbisWorld<br />

(Heckmann et al., 2005), since it allows to represent the user model in a very detailed structure, introducing<br />

29


information useful for the contextualization. Furthermore, as privacy can be a crucial issue in this kind of<br />

systems, UserML provides some tags to set privacy retention constraints.<br />

Whenever a MyC agent needs to answer to some questions from the student, if it can be connected to the VU-<br />

MAS, it contacts the VA of his advisor. Then MyC sends a request and the student profile to the VA, triggering<br />

its tasks. The VA answers sending back the information on the basis of the student profile received, attaching the<br />

needed resources. When the student asks some information, the VA looks for information asking to SAs. For<br />

example a SA can manage information about courses, such as name, teachers timetables and so on.<br />

MyC agent is supposed to be always connected to VU-MAS. However, if it is not possible to be online, or if the<br />

student decides to disconnect, MyC agent has to be able to give some suggestions. In this case, using the natural<br />

language generation strategies that will be shown in the next section, it gives an answer reusing information<br />

already downloaded on the device during previous sessions, combining also the cached animations and sounds.<br />

If these resources are not enough to answer the request, this is stored and the agent will give an answer when<br />

connected. Than, when offline, MyC agent can answer to questions that it has already answered to or that have<br />

some related links to previous requests, using its cache. In this way the interaction is much less engaging,<br />

because the agent cannot exploit new knowledge, but can be used to remember previous answers. This is an<br />

issue which will be overcome as well as the computational power increases, or broadband wireless connection<br />

become more spread. During the interaction MyCoach keeps tracks of all the questions and answers to update<br />

the student profile.<br />

MyCoach: a Personal Advisor Agent<br />

All the interaction described in the previous section is handled by MyCoach agent. This is an Embodied<br />

Conversation Agent, with the aim of giving the agent a more engaging look and feel. As mentioned in the<br />

introduction, MyCoach agent has been developed as an extension of the architectural schema developed in the<br />

context of the MagiCster project. In this project, the ECA was intended as an entity made up of two main<br />

components, a ‘Mind’ and a ‘Body’, interfaced by an I/O language, so as to overcome integration problems and<br />

to allow their independence and modularity.<br />

During the interaction, the Agent's Mind has the purpose of deciding what to communicate, by considering the<br />

dialogue history, the conversational context and the communicative goal related to the user request. The output<br />

of the mind module is an APML (Affective Presentation Markup Language) specification of the meaning to be<br />

communicated (De Carolis et al., 2003).<br />

The Body has the purpose to interpret and to render this output at the surface level, according to the available<br />

communicative channels: different bodies may have different expressive capabilities and therefore may use<br />

different channels. This approach was mainly driven by the definition of communicative function in Poggi et al.<br />

(Poggi et al., 2000). In their theory, a communicative function is a (meaning, signal) pair, where the meaning<br />

item corresponds to the communicative value of the signal item. For instance, a smile can be the signal of a<br />

“joy” emotion.<br />

Therefore, the Mind should convey only the meanings associated with the act to be communicated and the Body<br />

should interpret these meanings and render them into an expressive behaviour according to the interaction<br />

context and the device used in the interaction.<br />

In the first version of the architecture, the APML interpretation process was embedded in the Body Player (De<br />

Carolis et al., 2003). But, since the Agent’s believability is strictly related to features such as its personality and<br />

role and the cultural and social context, separating the way in which the Body will render that meaning enforces<br />

adaptivity .<br />

Mobile devices place limitations on the functionality of an embodied conversational agent since their<br />

computational and display capabilities are limited. Let’s see now how this architecture structure has been<br />

adapted for the development of an ECA that runs also on an handheld computer platform (Figure 2).<br />

30


Agent’s Mind<br />

Local<br />

Domain<br />

KB<br />

XAPML-SEML<br />

tables<br />

User Input: Communicative Goal<br />

SEML<br />

XAPML<br />

MIND<br />

Body Animation<br />

Generator<br />

SEML<br />

Wrapper<br />

Figure 2. Mind-Body Architecture<br />

Student profile<br />

The task of this module is to generate the specification of the Agent’s move at the meaning level. Planning is a<br />

task hard to compute, then it is not feasible to have a planner running on an handheld computer, moreover<br />

MyCoach has to run on and off-line. For this reason, we adopted a schema based approach.<br />

Given a communicative goal, derived by an interpretation of the user input, the Agent’s Mind selects from its<br />

plan library the schema or a combination of schemas that allow to satisfy that goal. The result is an XAPML<br />

(eXtended APML) specification of meanings to be communicated.<br />

APML is a markup language designed to represent the communicative functions (De Carolis et al., 2003). Each<br />

tag of this language is used to add some affective accent to the content of a sentence. For instance the<br />

tag is used to specify what is the communicative speech act of the sentence (Pelachaud and<br />

Poggi, 1998).<br />

The extension of APML includes the following information that were not present in the first version: i) besides<br />

the specification of what the agent has to communicate it is possible to specify the information background<br />

(access to more details, visualization of relevant domain objects, etc.); ii) the focus attribute of the<br />

tag allows to establish which is the main topic under discussion and to update the student profile<br />

with information about what has been communicated to the student; iii) the voice attribute has been added since,<br />

in case the student interact through its handheld device, the voice output has to be generated on the server side<br />

and then passed to the client for being played together with the body animation. These recipes have been<br />

designed on the basis of the results of a survey, in which the students were asked to rate a set of information<br />

about the department and student life. Figure 3 shows results of second year students.<br />

Analysing the survey results, major interests seem to concern information about modalities for choosing optional<br />

exams for completing the third year curricula, accessing student facilities, setting an examination test, get credits<br />

and recover from formative debits. Therefore, we developed XAPML schemas for answering to these families of<br />

questions and for getting more detailed answers once selected a topic question. These schema allows to organize<br />

the sequencing of the answer and to manage the multimodality of the interaction, combining text, voices, and<br />

expressions of the face.<br />

31


Nr of student<br />

25<br />

20<br />

15<br />

10<br />

5<br />

0<br />

a - where's the teacher's room<br />

b - information about department labs<br />

c - information about a course<br />

d - modality of examination<br />

e - personalized curricula<br />

f - where can i find the programs<br />

g - examination timetable<br />

h - date of degree session<br />

i - bureaucracy about thesis<br />

j - modality of degree sessions<br />

k - modality of registration<br />

l – deadlines<br />

m - office's location<br />

n – scholarship<br />

o - modules and forms<br />

p – stage<br />

q - suggestions about studying<br />

r - dates and rooms of seminars<br />

Figure 3. Results of the questionnaire of second year students<br />

s - how to acquire formative credits<br />

t - how to recover from formative debts<br />

u - how to record exams<br />

v - information about EDISU<br />

w - information about sport facilities<br />

x - colleges and restaurants<br />

y - student organized events<br />

z - examination rooms<br />

The following is a recipe for providing personalized suggestions about “optional courses”:<br />

<br />

<br />

[FORALL optional_course(topic-i) DO<br />

IF NOT(HasInterest(S,topic-i)) DO<br />

[<br />

I’m sorry for<br />

you but<br />

<br />

I have to suggest you to follow the course [topic-i]<br />

even if I know you don’t like<br />

[SuperClass(topic-i) OR Related(topic-i)]<br />

]<br />

IF (HasInterest(S,topic-i))<br />

I’m happy<br />

<br />

<br />

to suggest you to add the course of [topic-i] to your curriculum<br />

<br />

since I know that you like<br />

[SuperClass(topic-i)OR Related(topic-i)]<br />

]<br />

<br />

<br />

[FORALL optional_course(topic-i) DO<br />

IF detail [URI-topic-i]<br />

]<br />

<br />

<br />

These schemas are applied by the Mind in order to generate the instantiated XAPML specification. In the<br />

previous example, square parenthesis represent instructions to the XAPML generator that will substitute them<br />

with the appropriate text by manipulating information in the Domain Knowledge Base. This transformation is<br />

executed on the server side and not on the handheld computer where, as we will see later on, only instantiated<br />

XAPML specification are interpreted and rendered at a surface level.<br />

Agent’s Body<br />

modality of<br />

examination<br />

examination<br />

timetable<br />

formative credits<br />

sport facilities<br />

a b c d e f g h i j k l m n o p q r s t u v w x y z<br />

When the student downloads the agent from the Department Web Site she can choose its Body according to<br />

his/her preferences. The selected body, obviously, influences the way in which the agent is able to express<br />

meanings.<br />

32


Since our main aim was not to develop a new ECA player but to test the validity of our approach, at the moment<br />

we have developed only two types of bodies (Table 2): a face (male professor and a young female professor<br />

assistant) and a character (a funny ball). This last one could be used, for instance, for interacting with children.<br />

Each ECA conveys the same meanings using signals that are typical of its body. For instance the two faces will<br />

use as signal channels those typical of a face (eye, eyebrow, mouth, gaze, etc.) while the funny ball will use<br />

movements, colors and changes in its dimensions as communicative channels. The XAPML specification is not<br />

directly interpreted by the player that is the same for all the different bodies. This is possible since the proposed<br />

architecture decouples meanings from signals.<br />

Each body has a conditional meaning-signal table that allows to appropriately translate an XAPML tag into a<br />

Signal Expression Markup Language (SEML). SEML tags define the expressions that can be performed on each<br />

channel of the Body. Table 1 specifies how a body’s can be specified in SEML along the appropriate<br />

channels for the face (Gratch et al., 2002; Poggi et al., 2000) and the ball.<br />

Table 1. SEML signals expression for two different bodies<br />

Face Body Funny Ball<br />

<br />

<br />

<br />

<br />

I’m sorry for <br />

The professor:<br />

The funny ball:<br />

<br />

<br />

<br />

<br />


Vincenzo, a second year student, needs some help in choosing some optional exams to organize his<br />

study curriculum. His personal profile, besides his personal data, contains the information about<br />

his student curriculum (given exams and relative marks), interests and preferences. In particular,<br />

analysing these data, it is possible to infer that he is interested in Videogames and that he is<br />

skilled in Programming while he is not very good in Mathematics.<br />

Therefore, MyCoach has to suggest him the courses that, according to Vincenzo’s profile, are the most<br />

appropriate to his skills and previous years curriculum.<br />

Once the personal advisor is activated by the student, it greets him and prompts a list of tasks on which it could<br />

help (Figure 4a). From available options the student chooses the course planning help, by clicking on the relative<br />

link. The possible choices are planning of the official and personalized curricula. Let’s suppose that the student<br />

selects the personalized curriculum option (Figure 4b). Obviously, the student can insert a free text question<br />

using the keyboard icon if the list of options does not contain what he was looking for.<br />

Figure 4. MyCoach a) greeting the student, b) proposing help about completing the third year curriculum<br />

MyCoach will then contact the VA responsible for Vincenzo and will send to this agent the request and the<br />

statements in the student profile that are relevant to this goal. In this scenario the triggered communicative goal<br />

will be “Suggest(MyCoach,Vincenzo,Choose-Course)” and therefore the corresponding XAPML schema shown<br />

in the previous Section. This schema will be instantiated then with a list of courses that fit the user profile. To<br />

this aim, the VA asks to the SA responsible for curriculum planning the list of possible courses that will be<br />

extracted from the related KB.<br />

Courses<br />

I year II year III year<br />

SuperClass<br />

Prerequisite<br />

Related<br />

Compulsory Optional<br />

Video Games<br />

Mathematics<br />

Figure 5. Diagram describing the structure of the Courses KB.<br />

-----<br />

NM for Computer Graphics<br />

Interactive 3D Env<br />

34


Figure 5 illustrates a simple diagram representing the Courses KB. As we can see from the diagram, and<br />

considering the information in the student profile, a list of possible courses is determined. Thereafter, MyCoach<br />

suggests a list of possible courses to insert in the curriculum as shown in Figure 6.<br />

In the given scenario, the course Numerical Methods for Computer Graphics is considered useful, because it is a<br />

prerequisite for the course of Interactive 3DEnvironment, which is connected to the student interests in<br />

Videogames. However, since the student has got a bad mark in Mathematics, therefore this is not one of his<br />

preferred topics, the expression of the body of the agent for the first proposal is a sad face (Figure 6 a) and b)),<br />

while the second suggestion is communicated in a happy mode (Figure 6 c) and d)). The instantiated XAPML<br />

output generated for the illustrated example is the following:<br />

<br />

<br />

I’m sorry for<br />

you but<br />

I have to suggest you to follow the course of Numeric<br />

Methods for Computer Graphics <br />

even if I know that you<br />

don’t like Mathematics<br />

I’m<br />

happy<br />

to suggest you to add Interactive 3D Environment to your<br />

curriculum<br />

since I know that you<br />

really like VideoGames<br />

<br />

<br />

LINK<br />

LINK<br />

LINK<br />

LINK<br />

<br />

<br />

Figure 6 shows how the schema is played by the agent’s body when suggesting these courses to the student.<br />

Figure 6. a) and b) “happy-for” expression; c) and d) ”sorry-for” expression<br />

The student can always access his profile to modify his preferences or updating his curriculum clicking on an<br />

icon in the lower bar. When the user searches for details in a different page, it is always possible to go back and<br />

talk to the agent by selecting the icon on the bottom status bar of the handheld application. Moreover it is<br />

possible to change the attitude of the advisor, selecting different faces.<br />

35


Evaluation<br />

Recently embodied conversational agents have been widely used in e-learning projects, even if there are no<br />

proved evidence that the use of an ECA can improve the learning experience. To support our proposed approach<br />

we have developed a working prototype and we have conducted some evaluation studies with the main goal of<br />

assess the effectiveness of the system, in term of level of believability and reliability, perceived by the students.<br />

Moreover, this study was aiming also at assessing the student attitudes toward the agent in terms of how much<br />

the presence of an embodied agent improved the effectiveness and engagement of interaction. This has been<br />

evaluated through the use of a questionnaire aiming at understanding how much the student liked the character<br />

and how much they found helpful MyCoach suggestions.<br />

To this aim we have conducted a test with two groups of ten students of the second year of our Department. Both<br />

groups had to execute the task of planning their courses for the third year.<br />

Figure 7. User Profile interface<br />

The first group was asked to do it using a PDA and MyCoach personal advisor as interface. The second group<br />

was asked to perform the same task using the web pages of our Department. These are mainly a collection of<br />

printable files, so they can be considered similar to a paper guide.<br />

In order to give to MyCoach the capability to provide personalized suggestions we asked to the students<br />

belonging to the first group first to create their profiles using a simple user interface (Figure 7) and then to<br />

interact with MyCoach.<br />

trustworthiness<br />

easy of<br />

undestanding<br />

satisfaction<br />

likelihood<br />

2.5 2.75 3 3.25 3.5 3.75 4<br />

mean rating<br />

student guide<br />

agent<br />

Figure 8. Comparison of features between agent and students webpages<br />

36


After the experiment, a post-test questionnaire (Appendix 1) was administered to measure the impact of the<br />

suggestions about courses on participants’ subsequent choice. This comprised a range of different measures<br />

concerning both informational and interaction aspects. To assess the effectiveness of the ‘information provision’<br />

aspect of the message, we included ratings of satisfaction and ease of understanding (both on five-point unipolar<br />

Likert scales). In addition for the first group, questions about reliability and believability of the ECA were<br />

added.<br />

For comparing the data obtained from questionnaires we used t-test. As it could be seen from Figure 8, the level<br />

of trustworthiness is lower for MyCoach when compared to webpages of the department. The difference between<br />

means is statistically significant for p=0,05 where t-test value is t1(18)=-1,98. Moreover the ‘easy of<br />

understanding’ feature is higher for written text in the website. It is confirmed by t-test with t2(18)=-1,5.<br />

However difference between means becomes statistically significant for p=0,1. In our opinion this can be due to<br />

the fact that subjects had more time for reading written information whereas MyCoach advices were spoken.<br />

However, the grade of satisfaction and likelihood is higher for the agent. The t-test values are t3(18)=1,63 and<br />

t4(18)=2,69 respectively. The first feature is statistically significant at p=0,1 level and the second at p=0,05 level.<br />

In fact, the webpages of our department do not permit personalization, but contains a static list of course names<br />

without any link to student preferences or attitudes.<br />

As mentioned earlier, the questionnaire given to the first group of students contained a set of items to measure<br />

participants’ perceptions towards the agent. Participants were requested to rate the agent along the following<br />

dimensions: helpfulness, likeability, reliability and competency. All ratings were made using five-point unipolar<br />

likert scales.<br />

competent<br />

reliable<br />

likeable<br />

helpful<br />

3<br />

2,5 3 3,5 4<br />

mean rating<br />

Figure 9. Measure of some dimension of the agent behaviour<br />

The graph (Figure 9) shows that the reliability level has been evaluated with a lower score, even if the students<br />

considered the use of the personal agent helpful and likeable.<br />

Students think that the agent is helpful even if they showed to trust more paper based information. Unfortunately<br />

the reliability and trustworthiness of a system like this often relays on a long term experience. In our tests users<br />

interacted with the agent for a short period of time, which perhaps was not enough to create this relationship. In<br />

this view, improving the social and affective behaviour of the agent might lead to a higher level of reliability and<br />

trust.<br />

Implementation Issues<br />

VU-MAS is a MAS developed using JADE, a FIPA compliant framework to build multi agent system in java<br />

(JADE <strong>2006</strong>), and its extension LEAP for handheld devices. MyCoach is one of the agents in this MAS. The<br />

Service Agents can be either agents or web-services, which can be connected to the MAS using a gateway<br />

(Greenwood, Buhler & Reitbauer, 2005).<br />

MyCoach is implemented combining Macromedia Flash ActionScript with the .NET Compact Framework using<br />

Visual Basic.<br />

3,4<br />

3,8<br />

3,8<br />

37


The interaction may be performed on or off-line; when the user is on-line, the computation of the MIND output<br />

is performed server side and the rendering on the handheld (client-side). In this case, when the user connects to<br />

the server and asks for specific information the related communicative goal is passed to the system that computes<br />

and sends back the XAPML specification of the communication, the mp3 file corresponding to the voice output<br />

generated by a TTS on the server side and the set of frames needed for that animation if not yet available on the<br />

client side. When the agent is off-line uses pre-stored data to answer to questions, as described in the section<br />

about the architecture.<br />

On the client side, MyCoach composes and plays a combination of animation and sound resources loaded onto<br />

the Pocket PC.<br />

As in DESIA, with whom this system has a lot of similarities from the implementation point of view, we store<br />

the different animations for each body channel. A scene is a collection of sequences of animation frames that are<br />

composed in real time as needed according to the SEML specification. Obviously, one SEML specification is<br />

then translated into a sequence of frames that are synchronized with the mp3 files by the SEML-Wrapper<br />

Module.<br />

Conclusions<br />

MyCoach is a personal agent integrated into a Virtual University MAS. This is a system that has to support the<br />

students along their studies by providing orientation suggestions and information useful to solve their problems.<br />

The agents cooperate together to reach the main goal of finding useful information, which seems to be one of the<br />

most important issues to improve the student study career.<br />

MyCoach is an animated agent designed to be consulted on a handheld device. For this reason we decided to use<br />

an extension of the Mind-Body architecture already used in the context of the MagiCster project. In particular,<br />

the Mind, running on the server side, decides what to say and generates, through the use of plan “recipes”, a<br />

specification (in XAPML) of the meaning to be communicated by the Body that can be used to generate the<br />

agent behaviour on the handheld computer (client side). Since the architecture can support the use of different<br />

“bodies”, that can express meanings using different signals according to their available channels, a meaningsignal<br />

table specifies how a specific body has to convey particular meanings. This is represented through the use<br />

of another markup language SEML that specify which combination of signal to employ in correspondence of<br />

each XAPML tag. Then, the SEML specification is interpreted by a Wrapper, developed in Flash for MyCoach<br />

application, that is responsible for the synchronization and the rendering of the agent animations.<br />

We are aware that MyCoach agent animations are very simple, but this was a first step in order to test how the<br />

architecture could support this kind of interaction.<br />

The evaluation studies results are promising, so we are planning to develop more characters with different bodies<br />

and interaction attitudes in order to let a higher level of personalization. This is important not only for practical<br />

reason, but also because the more the agent can be personalized the more engaging the interaction could be. In<br />

particular, we plan to add to MyCoach mind the capability to use motivational and affective strategies to<br />

encourage the student when needed.<br />

Further evaluation studies will be done to investigate other features of the system. For example it could be<br />

interesting to compare the suggestions of our personal agent with those of a real advisor.<br />

References<br />

Bickmore, T. (2003). Relational Agents: Effecting Change through Human-Computer Relationships. PhD<br />

Thesis, Media Arts & Sciences, Massachusetts Institute of Technology.<br />

Bickmore, T. W., & Picard, R. W. (2005). Establishing and maintaining long-term human-computer<br />

relationships. ACM Transactions on Computer Human Interaction, 12 (2), 293-327.<br />

Cassell, J., Sullivan, J., Prevost, S., & Churchill, E. (2000). Embodied Conversational Agents, Cambridge, MA:<br />

MIT Press.<br />

38


Cassell, J. (2001). Embodied conversational agents: representation and intelligence in user interfaces. AI<br />

Magazine, 22 (4), 67-83.<br />

Cunha, M. M., Tavares, A. J., & Ferriera, L. (2005). Infrastructures for the Virtual University, EUNIS 2005 -<br />

Leadership and Strategy in a Cyber-Infrastructure World, 21-24 June 2005, Manchester, United Kingdom.<br />

De Carolis, B., Pelachaud, C., Poggi, I., & Steedman, M. (2003). APML, a Markup Language for Believable<br />

Behavior Generation. In H Prendinger and M Ishizuka (Eds), Life-like Characters. Tools, Affective Functions<br />

and Applications, Berlin: Springer.<br />

de Rosis, F., Cavalluzzi, A., Mazzotta, I., and Novielli, N. (2005). Can embodied conversational agents induce<br />

empathy in users? AISB'05 Virtual Social Characters Symposium, 12-15 April 2005, Hatfield, United Kingdom.<br />

de Rosis, F., De Carolis, B., Carofiglio, V.,& Pizzutilo, S. (2003). Shallow and inner forms of emotional<br />

intelligence in advisory dialog simulation. In H Prendinger & M Ishizuka (Eds.), Life-like Characters. Tools,<br />

Affective Functions and Applications, Berlin: Springer.<br />

Dehn, D. M., & van Mulken, S. (2000). The impact of animated interface agents: a review of empirical research.<br />

International Journal of Human Computer Studies, 52 (1), 1-22.<br />

Ekman, P. (1993). Facial expression of emotion. American Psychologist, 48, 384-392.<br />

FIPA Home Page (<strong>2006</strong>), http://www.fipa.org.<br />

Gratch, J., Rickel, J., Andre, J., Badler, N., Cassell, J., & Petajan, E. (2002) Creating Interactive Virtual<br />

Humans:Some Assembly Required. IEEE Intelligent Systems, July/August, 54-63.<br />

Greenwood, D., Buhler, P., & Reitbauer, A. (2005). Web Service Discovery and Composition using the Web<br />

Service Integration Gateway. Proceedings of the 2005 IEEE International Conference on e-Technology, e-<br />

Commerce and e-Service, 29 March – 1 April 2005, Hong Kong.<br />

Heckmann, D. (2003). Introducing Situational Statements as an integrating Data Structure for User Modeling,<br />

Context-Awareness and ResourceAdaptive Computing, ABIS 2003, Karlsruhe, Germany, 283-286.<br />

Heckmann, D., Schwartz, T., Brandherm, B., Kröner, A. (2005). Decentralized User Modeling with UserML and<br />

GUMO. Proceedings of the Workshop on Decentralized, Agent Based and Social Approaches to User Modelling<br />

(DASUM 2005), Edinburgh, Scotland, 61-65.<br />

JADE (<strong>2006</strong>). Java Agent Development Framework, http://jade.cselt.it/.<br />

Johnson, W. L., LaBore, L., & Chiu, Y. C. (2004). A Pedagogical Agent for Psychosocial Intervention on a<br />

Handheld Computer. AAAI Fall Symposium on Dialogue Systems for Health Communication, <strong>October</strong> 22-24,<br />

2004, Washington DC, USA.<br />

Marsella, S. C., Johnson, W. L., & LaBore, C. M. (2003). Interactive pedagogical drama for health interventions.<br />

In U. Hoppe et al. (Eds.), Artificial Intelligence in Education: Shaping the Future of Learning through Intelligent<br />

Technologies, Amsterdam: IOS Press, 341-348.<br />

Moreno, R., Mayer, R. E., Spires, H., & Lester, J. (2001).The case for social agency in computer-based teaching:<br />

Do students learn more deeply when they interact with animated pedagogical agents? Cognition and Instruction,<br />

19, 177-213.<br />

Pelachaud, C., Carofiglio, V., De Carolis, B., de Rosis, F., & Poggi, I. (2002). Embodied Contextual Agents for<br />

Information Delivery Applications. Proceedings of AAMAS'02, July 15-19, 2002, Bologna, Italy.<br />

Pelachaud, C. (2005). Multimodal expressive embodied conversational agents. In Proceedings of the 13th<br />

Annual ACM international Conference on Multimedia, ACM Press, New York, NY, 683-689.<br />

Pelachaud, C., & Poggi, I. (1998). Multimodal communication between synthetic agents. In Proceedings of the<br />

Working Conference on Advanced Visual interfaces, May 24-27, 1998, L'Aquila, Italy.<br />

39


Poggi, I., Pelachaud, C., & de Rosis, F. (2000). Eye communication in a conversational 3D synthetic agent. AI<br />

Communications, 13 (3), 169-181.<br />

Reeves, B., & Nass, C. (1996). The Media Equation, New York: Cambridge University Press.<br />

Roda, C., Angehrn, A., & Nabeth, T. (2001). Conversational Agents for Advanced Learning: Applications and<br />

Research. Proceedings BotShow 2001, June 14, 2001, Paris, France.<br />

W3C Web Services Architecture Working Group Note (2004). http://www.w3.org/TR/ws-arch/.<br />

Weiser, M. (1991). The Computer for the 21st Century. Scientific American, September, 94-104.<br />

40


Appendix 1<br />

Personal Information:<br />

a) Are you a student of : first year[ ] second year [ ] third year [ ]<br />

b) Age: ____ c) Sex: ____<br />

Please answer this questions:<br />

1) On the basis of what you have just heard how likely are you to follow the recommendations for<br />

planning your courses?<br />

1 2 3 4 5<br />

Not at all Extremely<br />

likely likely<br />

2) How satisfied were you with the suggestion contained in the message?<br />

1 2 3 4 5<br />

Not at all Extremely<br />

satisfied satisfied<br />

3) How easy did you think that the message was to understand?<br />

1 2 3 4 5<br />

Not at all Extremely<br />

easy easy<br />

4) How trustworthy did you think the content of the message was?<br />

1 2 3 4 5<br />

Not at all Extremely<br />

trustworthy trustworthy<br />

_________________________________________________________________________<br />

(Only for the group using the Agent Modality)<br />

For each word below, please indicate how well it describes the computer character that you have<br />

just seen. Note that you are now evaluating the character, NOT the message.<br />

a) Helpful<br />

1 2 3 4 5<br />

Not at all Extremely<br />

b) Likable<br />

1 2 3 4 5<br />

Not at all Extremely<br />

c) Reliable<br />

1 2 3 4 5<br />

Not at all Extremely<br />

d) Competent<br />

1 2 3 4 5<br />

Not at all Extremely<br />

End of questionnaire.<br />

41


Lanzilotti, R., Ardito, C., & Costabile, M. F., & De Angeli, A. (<strong>2006</strong>). eLSE Methodology: a Systematic Approach to the e-<br />

Learning Systems Evaluation. Educational Technology & Society, 9 (4), 42-53.<br />

eLSE Methodology: a Systematic Approach to the e-Learning Systems<br />

Evaluation<br />

Rosa Lanzilotti, Carmelo Ardito and Maria F. Costabile<br />

Dipartimento di Informatica, Università di Bari, 70125, Bari, Italy<br />

Lanzilotti@di.uniba.it<br />

Ardito@di.uniba.it<br />

costabile@di.uniba.it<br />

Antonella De Angeli<br />

School of Informatics, University of Manchester, PO BOX 88, M601QD, UK<br />

Antonella.de-angeli@manchester.ac.uk<br />

ABSTRACT<br />

Quality of e-learning systems is one of the important topics that the researchers are investigating in the last<br />

years. This paper refines the concept of quality of e-learning systems and proposes a new framework, called<br />

TICS (Technology, Interaction, Content, Services), which focuses on the most important aspects to be<br />

considered when designing or evaluating an e-learning system. Our proposal emphasizes user-system<br />

interaction as one of such important aspects. Guidelines that address the TICS aspects and an evaluation<br />

methodology, called eLSE (e-Learning Systematic Evaluation) have been derived. eLSE methodology<br />

combines a specific inspection technique with user-testing. This inspection, called AT inspection, uses<br />

evaluation patterns, called Abstract Tasks (ATs), that precisely describe the activities to be performed<br />

during inspection. The results of an empirical validation of the AT inspection technique, carried out to<br />

validate this technique, have shown an advantage of the AT inspection over the other two usability<br />

evaluation methods, demonstrating that Abstract Tasks are effective and efficient tools to drive evaluators<br />

and improve their performance.<br />

Keywords<br />

E-learning systems, Quality, Evaluation, Controlled experiment<br />

Introduction<br />

E-learning is becoming very important in fields where access to learning materials needs to be brought about<br />

effectively and efficiently. Its “any time, any place” nature could be a winning strategy for particular needs, such<br />

as decongestion of overcrowded education facilities, support for learners or lecturers who live far from schools<br />

and universities, life-long education. When making remote data and tools available to users it is necessary to<br />

consider their different characteristics, such as cultural background, technical experience, technological<br />

equipment, and physical/cognitive abilities. In the e-learning context, a major challenge for designers and<br />

Human-Computer Interaction (HCI) researchers is to develop software tools that can engage novice learners and<br />

support their learning even at a distance. Towards this end, there should be a synergy between the learning<br />

process and the learner’s interaction with the software. As for any interactive system, usability is a primary<br />

requirement. If an e-learning system is not usable, the learner spend more time learning how to use the software<br />

rather than learning the contents. Beside being usable, an e-learning system must be effective in meeting the<br />

instructor’s pedagogical objectives. System evaluation should thus integrate an assessment of the educational<br />

quality aspects of e-learning systems.<br />

Despite the large number of e-learning systems now available, one of the barriers to successful deployment of<br />

technology-based learning is the lack of high quality systems tailored to the needs of individual users and<br />

groups. Quality, from the Latin qualis, which means a pleasant thing, is an abstract term that assumes specific<br />

meanings according to the context in which it is used. From the end of the 1970s, in the software engineering<br />

context, some factors have been introduced as measures of the software quality. McCall affirms that quality<br />

factors represent attributes or characteristics of the software that a user or a client of the software couples with<br />

the quality of the software (McCall, 1994). Details on the first studies on quality factors can be found in<br />

(McCall, 1994; Boehm, 1978).<br />

When speaking of quality, it is important to consider the regulations for quality certification. In particular, the<br />

ISO/IEC 9126 establishes standards for ensuring the quality of a software product (ISO 9126, 1991),<br />

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the<br />

copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies<br />

are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by<br />

others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior<br />

specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.<br />

42


emphasizing that the quality is an attribute that depends on the users, the context, the goal, and the cost of the<br />

product.<br />

In the last few years, we have been involved in e-learning projects and faced several problems that impact<br />

learner-centred design, which presents its peculiarities with respect to general user-centred design (Quintana et<br />

al., 2001; Costabile et al., 2003; Ardito et al., 2004; Ardito et al., <strong>2006</strong>). The evaluation of e-learning systems<br />

deserves special attention, and evaluators need appropriate guidelines as well as effective evaluation<br />

methodologies (Zaharias et al., 2002). Unfortunately, the number of studies addressing evaluation of e-learning<br />

systems is relatively small and inadequate to the importance of the issue (Squires and Preece, 1999; Quinn et al.,<br />

2005). Moreover, it is often the case that the evaluation criteria are only vaguely stated (Parlangeli et al., 1999;<br />

Squires and Preece, 1999; Wong et al., 2003), so that an actual measurement of the system quality is left to<br />

subjective interpretation.<br />

In this paper, we propose a new framework for quality of e-learning systems, called TICS (Technology,<br />

Interaction, Content, Services), which focuses on the most important aspects to be considered when designing or<br />

evaluating an e-learning system. TICS makes explicit the importance of user-system interaction that is neglected<br />

by other authors, who implicitly assume that the user interface of an e-learning system does not influence user<br />

activities (Moore, 1989; Ewing and Miller, 2002). Actually, making the user interface ‘transparent’ is the<br />

ambition of any interactive system. In this way, users could concentrate only on their tasks. People who have<br />

experience in e-learning know the difficulties of users when interacting with an interface that is not usable. Many<br />

e-learning systems provide various functionalities and services, but it is often difficult, if not impossible, for<br />

users to find and use them in an effective and efficient way. Thus, our definition of e-learning systems quality<br />

highlights the importance of this aspect in designing and/or evaluating the overall quality of these systems. If<br />

designers are fully aware of the high value of the user interface, they will not neglect this aspect of e-learning<br />

systems and will create products of better quality, which will contribute to the success of technology-based<br />

learning.<br />

TICS aims at highlighting the main aspects that contribute to the quality of e-learning systems, so that designers<br />

and evaluators can focus on such aspects, thus providing what is required for good quality system. As a further<br />

contribution we have derived guidelines that address the TICS aspects and an evaluation methodology, called<br />

eLSE (e-Learning Systematic Evaluation). eLSE methodology combines a specific inspection technique with<br />

user-testing. This inspection aims at allowing inspectors that may not have a wide experience in evaluating elearning<br />

systems to perform accurate evaluations. It is based on the use of evaluation patterns, called Abstract<br />

Tasks (ATs), which precisely describe the activities to be performed during inspection. For this reason, it is<br />

called AT inspection. An empirical validation of the AT inspection technique has been carried out: three groups<br />

of novice inspectors evaluated a commercial e-learning system applying the AT inspection, the heuristic<br />

inspection, or user-testing. Results have shown an advantage of the AT inspection over the other two usability<br />

evaluation methods, demonstrating that Abstract Tasks are effective and efficient tools to drive evaluators and<br />

improve their performance.<br />

The paper has the following organization. First, contributions of some researchers in the domain of e-learning<br />

systems evaluation are reported. Then, the TICS framework is illustrated, highlighting the most important<br />

aspects to consider in designing and/or evaluating the overall quality of e-learning systems. Consequently, the<br />

eLSE methodology is presented and the validation of the AT inspection with its results is reported. Finally, the<br />

section Conclusion closes the paper.<br />

Quality of e-learning systems<br />

Various definitions and frameworks for the quality of e-learning systems are reported in literature, but the<br />

identified solutions appear as a short blanket, able to cover only some of the multiple aspects that characterize<br />

the complexity of the problem.<br />

Quality is defined as “the totality of characteristics of an entity that bear on its ability to satisfy stated and<br />

implied user needs” (ISO 9126, 1991). It includes a new quality model distinguishing between three different<br />

approaches to product quality:<br />

external quality, which is measured by the dynamic properties of the code when executed (such as response<br />

time)<br />

internal quality, which is measured by the static properties of the code, typically by inspection (such as path<br />

length)<br />

43


quality in use, which is measured by the extent to which the software meets the needs of the user in the<br />

working environment (such as productivity).<br />

External quality is the result of the combined behaviour of the software and the computer system, while quality<br />

in use is the effectiveness, productivity and satisfaction of the user when carrying out representative tasks in a<br />

realistic working environment. External measures can be used to validate the internal quality of the software.<br />

Quality in use measures the degree of excellence and can be used to validate the extent to which the software<br />

meets user needs. Appropriate internal attributes of the software are a pre-requisite for achieving the required<br />

external behaviour, and appropriate external behaviour is a pre-requisite for achieving quality in use (ISO 9126,<br />

1991).<br />

Chua and Dyson propose the ISO/IEC 9126 Quality Model (ISO 9126, 1991) to evaluate e-learning systems<br />

(Chua and Dyson, 2004). It provides an indication to educators and educational administrators of the quality of a<br />

system they are considering buying and provides a basis for comparison of different systems. However, for<br />

software developers without educational expertise, the ISO model alone would not be sufficient because it is a<br />

general software quality model and does not specify the particular teaching and learning activities needed for<br />

effective learning. Moreover, ISO/IEC 9126 presents some inherent weaknesses, particularly with regards to the<br />

usability characteristic. The authors suggest that this characteristic should be extended to include more specific<br />

factors such as consistency, simplicity, legibility. In addition, they propose the inclusion of user satisfaction as a<br />

global characteristic to summarise the general impact of the system on the users in their specific educational<br />

context (Chua and Dyson, 2004).<br />

Some authors have highlighted that quality in e-learning needs to be envisaged from different perspectives.<br />

However, they look at the quality primarily from a pedagogical point of view and emphasize the learner’s<br />

perspective (Ehlers, 2004), while they do not analyze in depth aspects related to the design of interactive<br />

software systems.<br />

Accordingly, Ehlers (2004) suggests that subjective quality requirements be structured in 7 fields of quality (see<br />

Figure 1):<br />

Quality Field (QF) 1 (Tutor Support) considers the preferences that learners have for communication and<br />

cooperation with the tutor of an online course.<br />

QF 2 (Cooperation and Communication in the Course) contains quality requirements for the course that<br />

learners express, that concern the communication and cooperation environment in which they work with<br />

other learners in learning groups, with experts or the tutor.<br />

QF 3 (Technology) is also important to learners. If technical requirements are fulfilled they do not raise the<br />

perceived quality very much - as they are taken for granted. Yet if the expected technical standards are not<br />

met the learners quality assessment decreases.<br />

QF 4 (Costs-Expectations-Benefits) refers to the information possibilities learners have about a course or the<br />

institution/organization which is offering the course.<br />

QF 5 (Information Transparency of Provider/Course) contains the provision of formal and standard<br />

information as well as individualized counseling on course contents, learning methodology or technical<br />

advice.<br />

QF 6 (Course structure) contains learners’ requirements concerning the structure of an e-learning course.<br />

Learners’ quality preferences clearly show that the presence of lessons as part of an e-learning course<br />

(blended learning) is of high importance to certain groups of learners, whereas others do not regard them as<br />

important.<br />

Finally, QF 7 (Didactics) covers aspects of content, learning goals, methods and materials. Experienced elearners<br />

are often very precise in their requirements concerning the didactical setting of an e-learning course.<br />

The quality fields are subdivided into 30 dimensions, each of them represents a set of criteria of learners’<br />

preferences that are clustered in a dimension on the basis of empirical evidence.<br />

As Ehlers, Rovinskyi and Synytsa have suggested to perform multidimensional quality evaluation according to<br />

the parameters identified as important by the learner (Rovinskyi and Synytsa, 2004). These parameters will be<br />

hereafter named quality metadata and will actually be a part of metadata for a course or derived from the<br />

metadata of its components. Quality metadata may be used to measure the correspondence of a course to the<br />

needs and expectations of a learner, thus determining the course’s quality from a learner’s perspective. Quality<br />

metadata fall into the following groups: learning goals (learning objectives and tasks, curricula and certificates);<br />

instructors and experts (their duties, qualification, and way of interaction); target audience (required knowledge<br />

and skills, prior education or professional background, learning groups and interaction); learning environment<br />

44


(required hard and software, additional plug-ins, complexity of environment, navigation, help, assessment<br />

instruments and feedback); learning resources (types of resources, availability on/off line, variety of<br />

representation, composition and complexity, types of activity offered, learning material completeness).<br />

Figure 1. Model of subjective quality requirements from [Ehlers, 2004]<br />

Other researchers have proposed to consider technological as well as educational aspects in evaluating e-learning<br />

systems. In (Herrington et al., 2001), a framework is presented as a checklist enumerating what are considered to<br />

be critical elements of quality learning environments. The checklist is based on the determination of critical<br />

elements within three main areas:<br />

the pedagogies area, referring to the activities that are fundamental for the learning process: the learning<br />

activities, proposed by the e-learning systems, would involve tasks that reflect the way in which the<br />

knowledge will be used in the real life settings; the learners would have the opportunity to work<br />

collaboratively; the lecturer’s role would be considered as a coach rather than an instructor; and, finally,<br />

authentic and integrated assessment tests should be used to evaluate learner’ achievement;<br />

the resources area, concerning the content and information provided to the learners: resources should be<br />

organized in ways that make them easily accessed. They should be current and based on literature reviews<br />

by lecturers and they should present various viewpoints to allow learners the opportunity to judge the merit<br />

of different arguments;<br />

the delivery strategies area, involving issues associated with the ways in which the course is delivered to the<br />

learners: the learning material should be accessible and available to all learners, including people with<br />

special needs, such as people with physical disabilities, or geographically isolated learners. Moreover, the<br />

learners should be count on the reliability of the technology: the e-learning system should allow learners to<br />

navigate and download materials within a reasonable period of time, and learners should be able to use<br />

various technologies to communicate and collaborate with each other and their teachers, and so on.<br />

In a publication that has been approved and publicized by the Italian Ministry of Education, Montedoro and<br />

Infante indicate three quality dimensions of e-learning systems: technology, content and services (Montedoro<br />

and Infante, 2003) Concerning technology, the educational process has to be supported by an on-line platform<br />

that manages the educational material. The platform has to be compatible with the international standards<br />

(AICC, SCORM, etc.) and to manage functions such as: analysis of the user profiles, building of personalized<br />

educational paths, the planning of the learning activities, the management of on-line courses and educational<br />

materials, interaction and in particular, forums, chat, virtual classrooms, and communications among learners.<br />

Regarding content, it is important to perform an accurate educational design that is learner-centered. It is<br />

characterized by the following elements: user orientation, the ease of navigation and fruition of the educational<br />

products and resources, interaction in all the phases if the educational process is enriched by specific and<br />

continuous feedback, length of the didactic unit not exceeding twenty minutes, content modularity, evaluation of<br />

the educational results. Finally, the services dimension increases the learner’s possibilities of contacting the<br />

tutor. The tutor has the task of managing the relations with the learners and guaranteeing rapid answers.<br />

Grutzner et al. state that the quality of e-learning systems is affected by four dimensions: the content of learning<br />

materials; the presentation of these materials; the pedagogic content, i.e., the way in which materials are taught;<br />

the overall functionality of the courseware. All four dimensions have to be considered at the same time and<br />

continuously throughout the courseware life-cycle to ensure high quality of the final product and thus to<br />

facilitate learning (Grutzner et al., 2004). The authors propose IntView, a systematic courseware development<br />

process, designed to support and integrate all of the different perspectives and views that are involved in the<br />

45


process of designing high quality courseware. The courseware engineering life-cycle model of the IntView<br />

methodology is accompanied by four engineering methods that assure the quality of the output artefacts of each<br />

phase. These methods include perspective-based inspections, prototyping, tests, and both formative and<br />

summative evaluations. Thus, the whole life-cycle is encompassed by verification and validation tasks that are<br />

assigned to different roles.<br />

In one of the early papers in distance education, Moore explicitly speaks of user interaction as an important<br />

aspect to be considered for the quality of e-learning systems and proposes a quality analysis model that considers<br />

three types of interaction (Moore, 1989). First of all learner-content interaction is considered, since it is the<br />

process of interacting with content that brings about changes in the learner's understanding and perspective.<br />

Learner-lecturer interaction is also important, because lecturers seek to stimulate the learner’s interest and to<br />

motivate the student to learn. Learner-learner interaction among members of a class is the third type of<br />

interaction, since it is often an extremely valuable resource for learning. Like Moore, Fata affirms that for<br />

evaluating the quality of e-learning systems it is important to consider three different types for interaction (Fata,<br />

2003): interaction between the learners and didactic material, the educational material just be of good quality,<br />

pleasant, easily usable, and equipped with an accurate and rich bibliography; interaction among learners, tutor,<br />

and experts: the tutor has to favour discussions among the learners, to reduce the learners’ sense of isolation.<br />

During the course, the learners must constantly be able to perceive the tutor’s presence; and during the<br />

conclusion of the course, the tutor must stimulate meta-cognitive reflection; and finally, interaction among<br />

learners: the communications among learners should take into account not only the discussions as group<br />

activities but also the free communications, in order to encourage a greater familiarity among learners. Other<br />

researchers have spoken of interaction as an important aspects for quality e-learning systems (Scanlon et al.,<br />

2000; Muirhead, 2001; Ewing and Miller, 2002), but it is worth noticing that they focus on the interaction<br />

between learner and educational content.<br />

All these types of interaction are mediated by the software system through which the content is delivered. As<br />

HCI researchers, we know the importance of designing the interface to allow a good interaction between the user<br />

and the provided content, which is the basis for effective learning. We must suggest models and techniques that<br />

support e-learning system designers in creating quality products.<br />

TICS framework for the quality of e-learning systems<br />

With respect to the other definitions of e-learning systems quality, we emphasize the interaction dimension and,<br />

specifically, the interaction between the user (teacher or learner) and the overall system, not only its content (the<br />

learning materials). On the basis of our experience, we believe that in the e-learning context, design<br />

methodologies and techniques have to ensure a user-system interaction that facilitates the learning process. It is<br />

not possible to neglect the influence of the user interface on the learner activities. Thus, the interface must be<br />

designed and evaluated on the basis of well defined criteria and methods, specific for e-learning (Lanzilotti,<br />

<strong>2006</strong>).<br />

Our novel definition says that “e-learning systems quality is the extent with which technology, interaction,<br />

content and offered services comply with expectations of learners and teachers by allowing them to learn/teach<br />

with satisfaction”.<br />

Figure 2. Quality aspects of e-learning systems<br />

46


By considering the literature on the e-learning systems quality (briefly reported in the previous section), the<br />

experience of human-computer interaction experts and the results of observing real users interacting with elearning<br />

systems, a new framework called TICS (Technology, Interaction, Content, Services), has been<br />

developed. It focuses on the most important aspects to be considered when an e-learning system is designed or<br />

evaluated (Figure 2). TICS primarily focuses on the quality dimensions recommended by the Italian Ministry of<br />

Education in (Montedoro and Infante, 2003), integrating them with the interaction dimension that, as we have<br />

discussed above, plays a crucial role in the fruition of the e-learning material. In the following each TICS aspect<br />

is described.<br />

Technology<br />

The technology aspect is specialized in the dimension hypermedial technology, that refers to the technological<br />

problems that can obstruct or make difficult the use of the e-learning system and to the hypermedia<br />

characteristics of e-learning systems i.e., compatibility with different operating systems; performance; monitor<br />

resolutions; hardware and software for system accessibility. By technology we also refer to the hypermedia<br />

characteristics of e-learning systems, such as selecting and integrating the media made available by current<br />

technology. In fact, when designing an e-learning system, we must accurately choose media that are really<br />

complementary and may enrich the contents without overloading the user’s perceptual and cognitive systems.<br />

Interaction<br />

The explicit inclusion of the interaction aspect is one of the novelty of our approach. We believe it is crucial for<br />

technology-based learning. This aspect involves presentation of the educational material and of the provided<br />

services and tools, and user activity performed during the interaction with the system. In particular, user errors<br />

should be prevented as much as possible. If an error occurs, the e-learning system should provide appropriate<br />

support to manage it. Learners not only interact with their educational material but they consider comments<br />

provided by other users. In fact, according to other researchers (Ewing and Miller, 2002; Moore, 1989), a great<br />

contribution to learning comes from interaction among learners and their lecturers. Researchers who neglect the<br />

importance of user-system interaction implicitly assume that the e-learning system interface does not influence<br />

user activities. People who have experience in e-learning know the problems that users have when interacting<br />

with an interface that is not usable. In some studies performed with users (Ardito et al., <strong>2006</strong>, Lanzilotti, <strong>2006</strong>),<br />

we have seen that if an e-learning system interface is not usable, learners spend more time learning how to use<br />

the software rather than learning the content, which is their main goal.<br />

Content<br />

The content aspect is more closely related to education. It refers to the educational process that happens during<br />

the interaction with the e-learning system. It focuses on the appropriateness and quality of the educational<br />

material that could be achieved through an accurate learner-centered design. This aspect also refers to the way<br />

the material is taught and to the capability of the e-learning system to propose study activities to the learner, who<br />

should also be free to autonomously choose his/her learning path (e.g. alternating moments of study, exercises,<br />

and verifications). Besides, the learner must have the possibility to go more deeply into the course content and<br />

the system must provide concrete, real, and helpful examples to facilitate understanding of the educational<br />

content.<br />

Services<br />

The services aspect refers to the application proactivity, that involves the tools that facilitate and support the user<br />

during the navigation through the system and the fruition of the educational material. It is necessary to provide<br />

users with communication tools, auto-evaluation tools, help services, search engines, references, scaffolding, and<br />

so on. Ease of learn and ease of use of such tools permit users to concentrate their efforts on the learning paths<br />

without being required to spend too much time trying to understand the way the e-learning system works.<br />

47


eLSE methodology<br />

Designers and evaluators can focus on TICS dimensions, for providing what is required for good quality system.<br />

As a further contribution we have derived guidelines that address the TICS aspects and an evaluation<br />

methodology, called eLSE (e-Learning Systematic Evaluation), which prescribes a structured flow of activities.<br />

The main idea of eLSE is that the most reliable evaluation results can be achieved by systematically combining<br />

inspection with user-based evaluation. Several studies have outlined how two such methods are complementary<br />

and can be effectively coupled to obtain a reliable evaluation process (Nielsen, 1993). In line with those studies,<br />

eLSE suggests coupling inspection activities and user-testing, and precisely indicates how to combine them to<br />

make evaluation more reliable and still cost-effective.<br />

eLSE derives from a usability evaluation methodology, called SUE (Systematic Usability Evaluation), originally<br />

developed for evaluating hypermedia systems (Matera et al., 2002). eLSE shares with SUE three important<br />

characteristics. First, eLSE couples inspection and user testing, to make the evaluation more reliable and still<br />

cost-effective. Each evaluation process starts by having evaluators inspecting the application and identifying<br />

possible problems and troubles. User testing is then conducted, whenever necessary, to validate some inspection<br />

findings with real users. Second, eLSE suggests to analyze an application along the TICS dimensions (i.e.<br />

hypermedial technology, presentation, user activity, educational process, application proactivity) that address the<br />

appropriateness of design with respect to the peculiar nature and purposes of the e-learning systems. Finally,<br />

eLSE proposes an inspection technique based on the use of Abstract Tasks (ATs) that are specifically defined for<br />

e-learning systems. For this reason, it is called AT inspection. ATs precisely describe which objects of the<br />

application to look for and which actions the evaluator must perform in order to analyze such objects.<br />

eLSE phases<br />

According to eLSE methodology, the activities in the evaluation process are organized into a preparatory phase<br />

and an execution phase. The preparatory phase is performed only once for each analysis dimension; its purpose<br />

is to create a conceptual framework that will be used to carry out actual evaluations. The output of the<br />

preparatory phase can be easily shared among different evaluators, or different evaluation laboratories that have<br />

similar interests and evaluate such applications from similar points of view. The preparatory phase consists of<br />

the identification of guidelines to be considered for the given dimensions and the definition of a library of ATs.<br />

The execution phase is performed every time a specific application must be evaluated. It mainly consists of<br />

inspection, performed by evaluators. If needed, inspection can be followed by user testing sessions, involving<br />

real users. At the end of each evaluation session, the evaluators must provide designers and developers with an<br />

organized evaluation feedback. The activities in the two phases are described in the following sections.<br />

The preparatory phase<br />

In the preparatory phase, a number of decisions must be taken and the definition of a specific set of Abstract<br />

Tasks must be carried out.<br />

Abstract Task formulation<br />

By considering TICS dimensions, the literature on e-learning, the experience of human-computer interaction<br />

experts, and the results of observing real users interacting with e-learning systems (Ardito et al., <strong>2006</strong>), a number<br />

of specific guidelines have been identified, to be taken into account during the initial design phase. Then, a set of<br />

Abstract Tasks focusing on these guidelines is identified to support the evaluators in their inspections; an AT<br />

addresses one or more TICS guidelines. In this way, the defined set of ATs allows inspectors to verify that all the<br />

design aspects that must be considered for developing a good e-learning system are taken into account.<br />

An AT is a description of what an evaluator has to do when inspecting an application. ATs guide the evaluator’s<br />

activities by describing which elements of the application to look for, and which actions the evaluators must<br />

perform in order to analyse such elements. In this way, even novice evaluators, with lack of expertise in usability<br />

and/or application domain, are able to come out with more complete and precise results. We use the term<br />

abstract task since it describes a task that an inspector performs when is evaluating an application, but this<br />

description is provided in general terms and abstracts from a specific application to be evaluated.<br />

48


ATs can be seen as evaluation patterns, making possible to maximize the reuse of evaluators’ know-how, by<br />

capturing usability inspection expertise, and by expressing it in a precise and understandable form, so that it can<br />

be easily reproduced, communicated, and exploited. They therefore allow evaluators to take “... advantage of any<br />

of the efforts done in previous works, to reduce the effort needed to achieve a new one” (Nanard et al., 1998).<br />

As stated above, ATs are precisely formulated by means of a template that provides a consistent format and<br />

includes the following items:<br />

AT Classification Code and Title: they univocally identify the AT, and succinctly convey its essence.<br />

Focus of Action: it shortly describes the context, or focus, of the AT, by listing the application components<br />

that are the evaluation entities.<br />

Intent: it describes the problem addressed by the AT and its rationale, trying to make clear which is the<br />

specific goal to be achieved through the AT application.<br />

Activity Description: it describes in detail the activities to be performed during the AT application.<br />

Output: it describes the output of the fragment of the inspection the AT refers to.<br />

Optionally, a comment is provided, with the aim of indicating further ATs to be applied in combination, or when<br />

available, significant examples of inspection findings should be reported, to better clarify which situations the<br />

evaluators should look for while applying the AT activity.<br />

ATs are defined by expert evaluators on the basis of their experience, complemented with observations of other<br />

experts performing various inspections. In this way, ATs are a means for capturing evaluators' expertise and for<br />

reusing it; moreover, they provide information about application domain, tasks and users, that is reported in the<br />

AT description. Being the heuristic evaluation based only on the use of heuristics, it lacks this information,<br />

which is usually provided through some training given to the evaluators prior to the inspection.<br />

As we already said, TICS is the framework to which eLSE also refers. Indeed, the defined ATs are classified in<br />

two main categories: Content learnability and Quality in use. Specifically, content learnability addresses both<br />

user activity and educational process. On the other hand, quality in use addresses hypermedial technology,<br />

presentation, and application proactivity. The ATs refer to ISO standards (ISO 9126, 1991; ISO 9241, 1997),<br />

thus supporting the evaluations of effectiveness, efficiency, security, productivity, and satisfaction.<br />

Table 1 reports an example of AT, whose title is “Graphical interface elements”. The AT derives from two<br />

guidelines referring to Presentation dimension, namely “Choose text dimension, font, and colour for having good<br />

readability” and “Avoid possible forms of distraction (e.g. flashing, sliding inscriptions…)”.<br />

Table 1. Example of AT<br />

QU_02: GRAPHICAL INTERFACE ELEMENTS<br />

Focus of action: interface graphical elements<br />

Intent: analyze the platform interface from the graphical viewpoint<br />

Activity description: Analyze:<br />

- the colours<br />

- the use of flashing or sliding inscriptions<br />

- the characters font and dimension<br />

- the coherence of the LO pages.<br />

Output: a list reporting if:<br />

- there is an exaggerated use of different colours<br />

- there is an exaggerated use of distraction form (flashing or sliding<br />

inscriptions)<br />

- characters are not easily readable<br />

- the different platform pages are not coherent among them.<br />

Our approach proposes an important distinction between the e-learning platform and the educational modules.<br />

The e-learning platform is the software environment that usually offers a number of integrated tools and services<br />

for teaching, learning, communicating, and managing learning material. The educational modules, also called<br />

Learning Objects, are the specific learning material provided through the platform. Design guidelines and ATs<br />

defined for the platform differ from those ones defined for e-learning modules, since different features and<br />

criteria need to be considered (Ardito et al., <strong>2006</strong>, Lanzilotti, <strong>2006</strong>).<br />

49


Execution phase<br />

Execution phase activities are carried out every time an e-learning system must be evaluated. They include two<br />

major jobs: a systematic inspection and a user-based evaluation.<br />

Systematic inspection is performed by evaluators. During the inspection, the evaluator uses the ATs to perform a<br />

rigorous and systematic analysis and produces a report in which the discovered problems are described, as<br />

suggested in the AT. The list of ATs provides a systematic guidance to the evaluator on how to inspect an<br />

application. Most evaluators are very good in analysing certain features of interactive applications; however,<br />

they often neglect some other features, strictly dependent on the specific application category. Exploiting a set of<br />

ATs ready for use allows evaluators with limited experience in a particular domain to perform a more accurate<br />

evaluation.<br />

User-based evaluation is conducted only when there is disagreement among the evaluators on some inspection<br />

findings, so that validation with real users becomes necessary. ATs are still useful since they indicate how to<br />

define the Concrete Tasks (CTs for short), i.e. the actual tasks that users are required to perform during the test.<br />

A CT is thus simply formulated by considering the activity description item of the AT whose application<br />

provided contrasting findings; this description is not general as in the AT but it explicitly refers to the application<br />

to be evaluated.<br />

Since the AT activity description is a formulisation of the user tasks, starting from this it is immediately possible<br />

to formulate experimental tasks which can guide users in the critical situations encountered by the evaluators<br />

during inspection. CTs are therefore conceived as a means of actually verifying the impact, upon the users, of the<br />

specific points of the application that are supposed to be critical for e-learning quality. In this sense, they make<br />

user-based evaluation better focused, so optimizing exploitation of the users resources and helping to obtain a<br />

more precise feedback for designers.<br />

During evaluation execution, a sample of users is observed while they are executing CTs and relevant data are<br />

collected (users’ actions, users’ errors, time for executing actions, etc.). The outcome of this is therefore a<br />

collection of raw data. In the result summary, these data are coded and organized in a synthetic manner and then<br />

analyzed.<br />

The last activity of the execution phase aims at providing the designers and developers of the application with an<br />

organised evaluation feedback. The result of this activity is an evaluation report describing the problems<br />

detected, possibly revised in the light of the user testing outcome, using the terminology provided in the AT for<br />

referring to system objects or interface elements, and for describing critical incidents. This standardised language<br />

increases the precision of the report and decreases the risk of misunderstandings.<br />

AT inspection validation<br />

The advantage of the AT inspection for e-learning systems over other evaluation techniques has been<br />

demonstrated by a controlled experiment. The study involved seventy-three senior students of a Human-<br />

Computer Interaction (HCI) class at the University of Bari in Italy. They were divided in three groups that were<br />

asked to evaluate a commercial e-learning system by applying the AT inspection, or the traditional heuristic<br />

evaluation, or a thinking aloud technique. Each group was assigned to one of the three experimental conditions.<br />

The heuristic inspection group had to perform an heuristic evaluation exploiting the “learning with software”<br />

heuristics (Squires and Preece, 1999). In the user testing group, every evaluator observed a student during the<br />

interaction with the e-learning application using the thinking aloud technique. Finally, the AT inspection group<br />

used the inspection technique with ATs proposed by eLSE methodology.<br />

In addition, we recruited 25 students of another computer science class, who use the e-learning system, thus<br />

acting as users for the user testing.<br />

A week before the experiment, all participants were given a 1-hour demonstration of the application to be<br />

evaluated. A few summary indications about the application content and the main functions were introduced,<br />

without providing too many details. A couple of days before the experiment, a training session of about one hour<br />

introduced participants with the conceptual tools to be used during the experiment. Each group participated in<br />

their specific training session.<br />

50


Data were collected in a group setting, but every participants worked individually. The study consisted of two<br />

experimental sessions lasting three hours each. During the first session, participants evaluated the e-learning<br />

system applying the technique they were assigned to. Participants of the three groups had to find usability<br />

problems in the application, and to record them on a booklet. A day after, each evaluator was asked to type their<br />

discovered problems in an electronic form. This was required in order to avoid readability problems during data<br />

analysis. In the electronic booklet, each participant reported the description of the problem, where it occurred,<br />

and how it was found. Finally, the evaluator gave a severity rating to the problem in the scale from 1 (I do not<br />

agree that this is a usability problem at all) to 5 (usability catastrophe).<br />

Results<br />

Two expert usability evaluators independently examined all the electronic forms in order to identify single and<br />

unique usability problems. This analysis led to the identification of 247 problems and 49 non problems, or<br />

statements which reported not understandable content or unverifiable information. Non-problems accounted for<br />

7% of the statements written by the inspectors applying the heuristic inspection, and 3% of the problems written<br />

by the inspectors applying the AT inspection or performing user testing.<br />

On average, participants reported significantly more problems when applying the AT inspection techniques<br />

(mean = 21), than when performing user testing (mean = 9.62) or the heuristic inspection (mean = 12.22). The<br />

difference is significant as demonstrated by the results of an Anova with evaluation technique as betweensubjects<br />

factor (F(2,70) = 25.45, p < .001). Post-hoc tests applying the Bonferroni’s correction indicated that the<br />

effect was due to the higher number of problems reported in the AT inspection condition, while there was no<br />

significant difference between heuristic inspection and user testing.<br />

An overall efficiency index was computed, which reflected the number of problems each participant found in 10<br />

minutes. The Anova indicated that overall efficiency is affected by experimental condition, F(2,70) = 3.80, p < .05.<br />

Post Hoc comparisons (Bonferroni’s correction) showed that eLSE was the most efficient technique (mean =<br />

1.19 problems in 10 minutes, standard error = .08). There were no differences between heuristics and user<br />

testing, in the order mean = .91, standard error = .08 and mean = .90, standard error = .08.<br />

The study also revealed that different techniques addressed different type of problems. User testing and heuristic<br />

inspection helped to highlight problems common to all interactive systems, whereas the AT inspection focused<br />

also on specific problems of e-learning. These findings confirm the peculiar characteristics of AT that<br />

specifically addresses the e-learning domain, provided that appropriate ATs are defined.<br />

Conclusions<br />

This paper has discussed the concept of the quality of e-learning systems. In particular, we have refined the elearning<br />

system quality by introducing the aspect of the user-system interaction that had been neglected by some<br />

researchers. There is the ambition that the system user interface is “transparent” to the user. Unfortunately,<br />

people, that have experience in e-learning, knows difficulties that user meets when s/he interacts with an<br />

unusable interface. From this derives the need to emphasize the interaction aspect and the usability of the user<br />

interface. Thus, we have defined a framework, called TICS, which focuses on the most important aspects to<br />

consider when an e-learning system is designed or evaluated.<br />

The paper have introduced an approach systematic to the evaluation of e-learning systems, called eLSE<br />

methodology, that prescribes a structured flow of activities. eLSE proposes an evaluation technique, called AT<br />

inspection. Such a technique falls into the category of the methods called “inspection” which do not involve<br />

users, but expert evaluators only. This inspection technique uses evaluation patterns, called Abstract Tasks,<br />

which make possible to maximize the reuse of the evaluator’s expertise and to express it in a precise and<br />

understandable form, so that it can be easily reproduced, communicated, and exploited.<br />

The results of the experiment performed to validate the AT inspection have shown that the inspection based on<br />

the use of ATs is capable to address specific issues of e-learning better than other techniques such as a<br />

specialised heuristic evaluation and user testing.<br />

During the experiment, the students expressed positive opinions on the use of ATs. They said that the guidance<br />

offered by ATs allows them to perform the inspection more accurately. Students perceived to have more control<br />

51


on the inspection process and, consequently, they were confident to obtain good results. Furthermore, the two<br />

experts that have examined the electronic forms filled by the inspectors noticed that ATs enforce standardization<br />

and uniformity of inspection results, since all inspectors are guided by the output item writing their evaluation<br />

report.<br />

We already defined a good number of ATs addressing the quality issues of e-learning systems. Other ATs are<br />

currently under development that consider the overall user experience in the e-learning domain.<br />

References<br />

Ardito, C., De Marsico, M., Lanzilotti, R., Levialdi, S., Roselli, T., Rossano, V., & Tersigli, M. (2004). Usability<br />

of E-Learning Tools. Proceedings of AVI 2004, May 25-28, 2004, Gallipoli, Italy, 80-84.<br />

Ardito, C., Costabile, M.F., De Marsico, M., Lanzilotti, R., Levialdi, S., Roselli, T., & Rossano, V. (<strong>2006</strong>). An<br />

Approach to Usability Evaluation of e-Learning Applications. Universal Access in the Information Society<br />

International Journal, 4 (3), 270–283.<br />

Boehm, B. (1978). Characteristics of Software Quality, New York: North Holland Publishing.<br />

Chua, B. B., & Dyson, L. E. (2004). Applying the ISO 9126 Model to the Evaluation of an e-Learning System.<br />

Proc. of ASCILITE 2004, December 5-8, Perth, Australia, 184-190.<br />

Costabile, M.F., De Angeli, A., Roselli, T., Lanzilotti, R., & Plantamura, P. (2003). Evaluating the Educational<br />

Impact of a Tutoring Hypermedia for Children. Information Technology in Childhood Educational Annual, 289-<br />

308.<br />

Ehlers, U. (2004). Quality in e-Learning form a Learner’s Perspective, European Journal of Open Distance and<br />

E-Learning, retrieved July 30, <strong>2006</strong> from<br />

http://www.eurodl.org/materials/contrib/2004/Online_Master_COPs.html.<br />

Ewing, J., & Miller, D. (2002). A Framework for Evaluating Computer Supported Collaborative Learning.<br />

Educational Technology & Society, 5 (1), 112-118.<br />

Fata, A. (2003). La Qualita' nella Formazione a Distanza. Psico-Pratika, 4, retrieved April <strong>2006</strong> from<br />

http://www.humantrainer.com/articoli/anna_fata_qualita_formazione_a_distanza.html.<br />

Grutzner, I., Weibelzahl, S., & Waterson, P. (2004). Improving Courseware Quality through Life-Cycle<br />

Encompassing Quality Assurance. Proc. of Symposium on Applied Computing (SAC’04), March 14-17, 2004,<br />

Nicosia, Cyprus, 946-951.<br />

Herrington, A., Herrington, J., Oliver, R., Stoney, S., & Willis, J. (2001). Quality Guidelines for Online Courses:<br />

The Development of an Instrument to Audit Online Units, Proc. of ASCILITE 2001, December 9-12, 2001,<br />

Melbourne, Australia, 263-270.<br />

ISO 9126 (1991). ISO/IEC 9126: Information technology - Software Product Evaluation - Quality characteristics<br />

and guidelines for their use.<br />

ISO 9241 (1997) ISO 9241: Ergonomics Requirements for Office Work with Visual Display Terminal (VDT) -<br />

Parts 1-17.<br />

Lanzilotti, R. (<strong>2006</strong>). A Holistic Approach to Designing and Evaluating e-Learning Quality: Usability and<br />

Educational Effectiveness, PhD dissertation, Dip. Informatica, Università di Bari, Bari, Italy, <strong>2006</strong>.<br />

Matera, M., Costabile, M.F., Garzotto F., & Paolini P. (2002). SUE Inspection: an Effective Method for<br />

Systematic Usability Evaluation of Hypermedia. IEEE Transactions on Systems, Man and Cybernetics - Part A,<br />

32 (1), 93-103.<br />

McCall, J. A. (1994). Quality factors. In John J. Marciniak, (Ed.), Encyclopaedia of Software Engineering, New<br />

York: John Wiley, 958-969.<br />

52


Montedoro, C., & Infante, V. (2003). Linee Guida per la Valutazione di Qualità del Software didattico nell'e-<br />

Learning: ISFOL, I libri del Fondo Sociale Europeo, Roma.<br />

Moore, M.G.(1989). Three Types of Interaction. The American Journal of Distance Education, 3 (2), 1–6.<br />

Muirhead, B. (2001). Interactivity Research Studies. Educational Technology & Society, 4 (3), 108-112.<br />

Nanard, M., Nanard, J., & Kahn, P. (1998). Pushing Reuse in Hypermedia Design: Golden Rules, Design<br />

Patterns and Constructive Templates, Proc. of Hypertext ’98, the 9 th ACM conference on Hypertext and<br />

hypermedia: links, objects, time and space, June 20-24, 1998, Pittsburgh, PA, USA, 11-20.<br />

Nielsen, J. (1993). Usability Engineering, Cambridge, MA: Academic Press.<br />

Parlangeli, O., Marchigiani, E., & Bagnara, S. (1999). Multimedia System in Distance Education: Effects on<br />

Usability. Interacting with Computers, 12, 37-49.<br />

Quinn, C.N., Alem, L., & Eklund, J. (2005). A pragmatic evaluation methodology for an assessment of learning<br />

effectiveness in instructional systems, retrieved July, <strong>2006</strong> from,<br />

http://www.testingcentre.com/jeklund/Interact.htm.<br />

Quintana, C., Carra, A., Krajcik, J., & Soloway, E. (2001) Learner-Centred Design: Reflections and New<br />

Directions. In Carroll (Ed.), Human-Computer Interaction in the New Millennium, New York: Addison-Wesley,<br />

605-626.<br />

Rovinskyi, D., & Synytsya, K. (2004). Distance Courses Quality: A Learner’s View. Proc. of Fourth IEEE<br />

ICALT'04, August 30-September 1, 2004, Joensuu, Finland, 1080-1081.<br />

Scalon, E., Jones, A., Barnard, J., Thompson, J., & Calder, J. (2000). Evaluating Information and<br />

Communication Technologies for Learning. Educational Technology & Society, 3 (4), 101-107.<br />

Squires, D., & Preece, J. (1999). Predicting quality in Educational Software: Evaluating for Learning, Usability,<br />

and the Synergy between them. Interacting with Computers, 11 (5), 467-483.<br />

Wong, B., Nguyen, T. T., Chang, E., & Jayaratna, N. (2203). Usability Metrics for E-Learning. Lecture Notes in<br />

Computer Science, 2889, 235–252.<br />

Zaharias, P., Vasslopoulou, K., & Poulymenakou, A. (2002). Designing On-Line Learning Courses:<br />

Implications for Usability, retrieved July <strong>2006</strong>, from http://www.japit.org/zaharias_etal02.pdf.<br />

53


Piccinini, N., & Scollo, G. (<strong>2006</strong>). Cooperative Project-based Learning in a Web-based Software Engineering Course.<br />

Educational Technology & Society, 9 (4), 54-62.<br />

Cooperative Project-based Learning in a Web-based Software Engineering<br />

Course<br />

Nicola Piccinini<br />

Università di Verona, Dipartimento di Informatica, Ca' Vignal 2, Strada Le Grazie 15, I-37134 Verona, Italy<br />

piccinini@sci.univr.it<br />

http://www.sci.univr.it/~piccini<br />

Giuseppe Scollo<br />

Università di Catania, Dipartimento di Matematica e Informatica, Viale Andrea Doria, 6, I-95125 Catania, Italy<br />

scollo@dmi.unict.it<br />

http://www.dmi.unict.it/~scollo<br />

ABSTRACT<br />

Even in self-organized project-based learning, the instructors’ role re-mains critical, especially in the initial<br />

orientation provided to the students in order to grasp the educational goals and the various roles they may<br />

undertake to achieve them. In this paper we survey a few questions proposed to that purpose in a web-based<br />

software engineering course, together with relevant answers, we outline the project set-up methodology<br />

aimed at providing students with that initial orientation in the laboratory part of the course, we collect a few<br />

empirical data out of the latest seven-year history of the course and, finally, we put the presented work in<br />

the context of current approaches to software engineering education and draw brief conclusions.<br />

Keywords<br />

Problem-based learning, Project-based learning, Cooperative web-based learning, Self-organized learning,<br />

Meta-cognitive reinforcement<br />

Introduction<br />

Since seven years, the Software engineering course offered at the CS department in Verona relies upon the use of<br />

a cooperative web-based organization, whereby laboratory project work as well the course organization itself<br />

evolve following feedback and proposals made by the students. The educational approach thus matches the<br />

profile of “problem-based learning, project-based learning, and collaborative problem solving” Nesbit & Winne<br />

(2003). Furthermore, besides using a web-based cooperation platform BSCW (2005) to support interaction,<br />

coordination and resource sharing, the web itself is used as a source of knowledge exploration and support to<br />

collaborative inquiry learning (Chang et al., 2003; Salovaara, 2005), within a largely self-organized project<br />

teamwork.<br />

Nonetheless, the instructors’ role remains critical, especially in the initial orientation provided to the students in<br />

order to grasp the educational goals settled from the outset, and the various roles and responsibilities they may<br />

undertake to achieve those goals. This paper is aimed at corroborating this statement with empirical evidence<br />

drawn from the subject course experience. This is presented according to the following organization of the paper.<br />

In the next section, we survey a few questions presented to students in the initial part of the course, to the<br />

aforementioned orientation purpose. In the subsequent section, a select blend of answers proposed to those<br />

questions are reviewed and discussed, with reference to the main statement of the paper. We then proceed to<br />

outline the project set-up methodology that is aimed at providing students with initial orientation in the<br />

laboratory part of the course. Empirical data out of the latest seven-year history of the course laboratory are then<br />

collected and discussed, again with reference to the main statement. We finally put the presented work in the<br />

context of current approaches to software engineering education and draw brief conclusions. A more extensive,<br />

albeit earlier (hence less up-to-date) report on this web-based laboratory learning experience is available (in<br />

Italian) in Piccinini & Scollo (2005).<br />

Questions<br />

The following questions are excerpted (and translated from Italian) from the on-line lecture notes, which can be<br />

consulted at Scollo (2005).<br />

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the<br />

copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies<br />

are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by<br />

others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior<br />

specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.<br />

54


Q.1: Use of software engineering principles in web design<br />

The notes start with an introductory lecture where 1) several analogies are pointed out between software design<br />

and web design, and 2) traditional software engineering principles are presented and argued about. The very first<br />

group of exercises then looks like as follows.<br />

We propose a few examples of web design problems, all somehow connected to its rapid evolution<br />

character (...) whereas we propose as exercises: 1) the identification of software engineering<br />

principles, out of those introduced in the lecture notes, that appear relevant to each of the problems<br />

proposed, and 2) the development of a rationale for their respective applications to evaluation of<br />

solutions to those problems.<br />

With websites whose management brings frequent changes, users find it useful to get information<br />

about website updates, that is, answering the question: What’s new? Identify and evaluate at least<br />

two different solutions to the problem of how to provide this service.<br />

etc.<br />

Q.2: Inquiry into ergonomic qualities of the course website<br />

In a subsequent lecture, software engineering principles are put in the wider context of quality design rules, some<br />

of which are drawn from ancient, philosophical tradition (such as Ockham’s Razor, Duns Scott’s Consistency<br />

Rule, etc.). Use of these rules is then invited to be exercised through inquiry into ergonomic qualities of the<br />

course website. To this purpose the following questions are proposed, among others:<br />

Find aspects (of the structure and/or contents) of the website of this course which violate some<br />

(which?) of the proposed quality design rules, and motivate your answer.<br />

Find aspects (of the structure and/or contents) of the website of this course which satisfy some<br />

(which?) of the proposed quality design rules, and motivate your answer.<br />

These questions, as well as those relating to subsequent lectures, share a common premiss, which alludes at<br />

expected students’ contributions as co-designers of the course website contents:<br />

We do not propose to consider design qualities of the BSCW system as such, but only qualities<br />

that be relevant to the use of this system for the educational purposes of this course. The following<br />

may thus come into play:<br />

aspects of the BSCW system as such, that prove relevant to the aforementioned use;<br />

aspects of its use made by the designers of this workspace (instructors and tutors), who provide a<br />

web-based educational service to the course participants, and so are both system users and service<br />

providers at the same time;<br />

aspects of the use of this system and of this workspace made by the users of the afore-mentioned<br />

educational service, who, too, contribute to modify the structure and contents of the workspace,<br />

hence also have their share of responsibility for the ergonomic quality of the website in question.<br />

Q.3: Self-referential inquiry into workgroup organization<br />

As a third, final case of interest we excerpt a couple of questions relating to workgroup organization, that are<br />

based on a preliminary working assumption which takes the flavour of a self-referential case study:<br />

Consider the following problem as a working assumption:<br />

Organize (part of) the documentation that is produced by the participants to this edition of<br />

the course, in the form of a website that would be suitable for use by the instructors and<br />

participants to future editions of the same course.<br />

With reference to such a working assumption:<br />

1. propose working practices meant to help the efficiency of the production process;<br />

2. characterize an organizational structure that would prove both adequate to the objectives and<br />

compatible with the production context constraints;<br />

3. etc.<br />

55


Answers and reinforcement<br />

A.1: Use of software engineering principles in web design<br />

The first questions listed in the previous section were proposed since the 2002 edition of the course. Following<br />

students’ feedback, viz. a first series of “systematically wrong” answers, their presentation is now endowed with<br />

a supplement of meta-cognitive information, just aimed at clarification and reinforcement of the stated goal.<br />

Here is what is added to this purpose, and why. The first answers only dealt with the proposed web design<br />

problems, with no reference to software engineeering principles whatsoever. An example of such kind of<br />

answers is the following:<br />

The first method to handle such a situation consists in mailing an update report, after every update,<br />

where the various changes made are described and documented.<br />

The second proposal is to display a list of the recent changes as initial page.<br />

This led to inclusion of the following premiss to the exercise:<br />

The use of software engineering principles, such as those proposed in the lecture notes, proves<br />

purposeful to identification as well as evaluation of solutions to web design problems. Each of the<br />

proposed exercises asks to validate this statement in a concrete problem. What is thus required is<br />

not so much to invent a solution to each of the proposed problems as, rather, to reason about the<br />

use of the aforementioned principles to those purposes.<br />

A.2: Inquiry into ergonomic qualities of the course website<br />

What makes the following answers interesting, from an educational process viewpoint, is that they reveal a<br />

limited inquiry into the proposed subject of investigation. This gives instructors the opportunity to point at<br />

further inquiry directions and to interactively refine the students’ knowledge and understanding, again with a<br />

meta-cognitive reinforcement side-effect. Here are two examples, respectively referring to the two questions<br />

quoted in the previous section, where a student’s answer is excerpted, followed by the instructor’s reply.<br />

A.: In my opinion an important functionality is lacking within the BSCW system, and this may<br />

violate the completeness rule. The functionality which is not present (or I didn’t find) in the<br />

BSCW is that which would enable one to download a whole folder, including its contents.<br />

R.: The functionality is available, by means of the creation of a compressed archive of the folder,<br />

both for download and for upload.<br />

Note that the student is well aware that s/he may have failed to locate the desired function. The instructor’s reply<br />

thus works as a solution advice to a student’s inquiry problem.<br />

A.: The very structure of the BSCW system warrants conformance to the orthogonality rule. The<br />

organization into folders and subfolders, each of them dealing with a particular subject, allows the<br />

separation of distinct aspects and a quicker information retrieval by the user.<br />

R.: Rather than “warrants” I should like to say “enables”, since it’s up to the content producers to<br />

actually maintain separation of independent aspects by means of their placement into distinct<br />

folders.<br />

In this case, the instructor’s reply reinforces the alluded students’ responsibility as content producers.<br />

A.3: Self-referential inquiry into workgroup organization<br />

A seemingly surprising fact about the third group of questions recalled in the previous section is that none of<br />

them has ever been answered directly so far. This can be explained as follows. Unlike the previous questions,<br />

which are proposed at an early stage of the course, the subject questions occur at an advanced stage, when<br />

laboratory project work is well beyond take-off. Answering the proposed questions is an optional assignment in<br />

all cases, but with the first, early questions there’s no alternative assignment anyway. On the contrary, at the<br />

advanced stage when the subject of group organization comes to be dealt with, practice on it may well take place<br />

within the self-organized context of each project teamwork.<br />

56


The proposed questions prove useful nonetheless, in that they inspire similar questions which groups do address<br />

within their own self-regulated contribution to the laboratory production work, thereby answering those<br />

questions indirectly, viz. with reference to a different, albeit related working assumption. Rather than working as<br />

task assignments, thus, questions of this kind act as models for laboratory project set-up and organization, which<br />

is the subject of the next section.<br />

The lack of direct answers to the proposed questions, together with their proven usefulness in the development of<br />

student projects are thus no failure but rather to be taken as a sign of success of a scaffolding strategy, whereby<br />

instructors first model the desired working skills to be developed and then gradually shift responsibility to the<br />

students.<br />

Laboratory project set-up<br />

In order to accomplish the laboratory course tasks, students have got to put the concepts and methods learnt from<br />

the lectures into practice. This is organized by way of simulation of a (fairly liberal) software production<br />

environment, where project ideas and proposals are invited, project teams are dynamically created around them,<br />

and team projects are viewed as mildly co-ordinated constituents of a somewhat larger “laboratory project”,<br />

where industry-like control and co-ordination mechanisms are experimented.<br />

A project idea proposal does not only involve a concise description of the functionality of an intended software<br />

product, but is also required to exhibit a clear identification of the pursued educational objectives, mostly<br />

specified in the form of a list of intended project deliverables. Here it is understood that the production of a<br />

certain deliverable, say a requirements specification, a risk management plan, or a collection of architectural<br />

models, is a task that is instrumental to learn how to apply concepts and methodological prescriptions relating to<br />

that kind of task—this educational objective is thus designated by the corresponding deliverable.<br />

The intended software product is by no means expected to be actually produced, only the planned deliverables so<br />

are. However, in addition to setting educational objectives, a project idea proposal may include quality<br />

objectives. These refer to the intended software product, are kept distinct from the educational objectives, and<br />

denote the intent to evaluate and document the influence of desired quality characteristics of the specific product<br />

on the production process to be planned.<br />

Team projects are largely self-organized, in that all project activities, ranging from project idea formulation and<br />

choice of educational objectives through project team composition, production organization and actual<br />

execution, are left to the students. Quality of the organization work is a most relevant subject of the final<br />

evaluation, which is in principle aimed at evaluating the production process maturity rather than quality of the<br />

resulting products.<br />

Despite full-range self-organization, instructors do play a key role of guidance and control, which shows up at<br />

least on three subsequent stages of project evolution:<br />

1. Initial explanation, of what we are summarizing in this section, in classroom and by means of on-line<br />

documentation, in particular Piccinini (2005).<br />

2. Team project set-up. Before starting each team project work, students are strongly encouraged to seek<br />

instructors’ advice through a public on-line discussion about their project idea. Other students may of course<br />

join the discussion. This starts with a project idea proposal, hopefully including the pursued educational<br />

objectives, and sometimes also including a first identification of team members. The instructor may ask for<br />

further clarification when any of the following is deemed to hold:<br />

the idea is not clearly expressed,<br />

the educational objectives are inaccurate, confuse or (as it often happens) missing,<br />

the quality objectives, if present, are inaccurate or confuse.<br />

The discussion continues until no doubt is left and the team is formed.<br />

3. Final exam evaluation. This relates to the whole course program, but draws upon the project work developed<br />

in the laboratory.<br />

The second out of the previous three points is the most important one in the laboratory course organization.<br />

Since every discussion is publicly held on-line, it gives rise to an interesting case study for course students—<br />

those of the current edition as well as those of future ones. Throughout the latest seven-year course history, those<br />

discussions have come to build-up a fairly large information corpus that, besides comprising a rich bestiary of<br />

57


potential project ideas (from pure fictionary ones to potential merchandisable products), proves amenable to<br />

quantitative analysis, such as that which is presented in the next section.<br />

Laboratory project evolution, collected data samples<br />

The data sample sizes are displayed in Table 1: a total of almost 1000 students attended the seven course<br />

editions.<br />

Table 1. Data sample sizes<br />

edition students ideas projects % of ideas<br />

1999-2000 65 12 12 100%<br />

2000-2001 54 14 13 93%<br />

2001-2002 167 38 33 87%<br />

2002-2003 185 45 43 96%<br />

2003-2004 187 42 38 90%<br />

2004-2005 143 31 30 97%<br />

2005-<strong>2006</strong> 174 41 40 98%<br />

Total 975 223 209 94%<br />

Table 2. Project idea discussions<br />

edition 1999-2000 2000-1 2001-2 2002-3 2003-4 2004-5 2005-6 Total<br />

# ideas 12 14 38 45 42 31 41 223<br />

correct 5 5 11 15 21 7 13 77<br />

42% 36% 29% 33% 50% 23% 32% 35%<br />

clarified 1 2 6 6 4 2 7 28<br />

8% 14% 16% 13% 10% 6% 17% 13%<br />

refined 0 4 8 8 7 7 13 47<br />

0% 29% 21% 18% 17% 23% 32% 21%<br />

edu.oriented 0 7 20 24 12 17 19 99<br />

0% 50% 53% 53% 29% 55% 46% 44%<br />

qual.oriented 0 0 1 4 8 9 6 28<br />

0% 0% 3% 9% 19% 29% 15% 13%<br />

team set-up 1 2 6 10 11 7 13 50<br />

8% 14% 16% 22% 26% 23% 32% 22%<br />

unfinished 1 2 8 9 6 8 2 36<br />

8% 14% 21% 20% 14% 26% 5% 16%<br />

The data relating to the current course edition, 2005–<strong>2006</strong>, should not be considered as final ones, since course<br />

exams are on schedule for the rest of the year; students are not required to physically attend lectures, as they may<br />

get the course materials as well as organize team projects on-line. They proposed 223 different project ideas;<br />

most of these (209, nearly 94%) evolved to a concrete project, run by a working team with well defined<br />

objectives.<br />

However, in the majority of the sample cases, the transition from project idea to actual project is not immediate.<br />

Table 2 shows, in actual as well as percentual value, the incidence of the most frequent cases of delayed project<br />

start-up, as found in the project idea discussions, where corrections or refinements of idea proposals were put<br />

forward. This yields a quantitative measure of the relevance of the instructor's role to orientate the project startup<br />

along the desired educational path. The first row in Table 2 lists the number of project idea proposals, briefly<br />

“ideas”, per course edition. Subsequent rows are labelled according to the following terminology.<br />

correct are the ideas that appear sufficientely clear, accurate and complete since their first formulation. For<br />

example, students who débute with:<br />

Our idea is to design a on-line store of CD's, DVD's and, in general, consumable electronic<br />

materials (printer cartdridge, etc.). The customer may consult the on-line catalogue and book the<br />

chosen products. There will also be a forum where the customer may comment on the purchased<br />

goods.<br />

58


The project objectives are:<br />

quality control (functionality, reliability, usability, maintainability),<br />

feasibility study,<br />

risk management.<br />

Group components: [omissis]<br />

All is right and nothing is missing, hence the instructor replies:<br />

Good job, go ahead!<br />

clarified are the correct ideas that receive either additional suggestions from the instructor or requests for further<br />

information from the instructor and/or the students.<br />

refined are those ideas whose initial description is too vague, inaccurate, or even confuse, for example:<br />

We intend to design a software for the management of the civil guard emergency units. Units coordination<br />

will be driven by the emergency type and by the skills and availability of staff<br />

members.<br />

Although the use context of the target software is well described, an outline of the desidered functionality is<br />

missing, and this makes it difficult to evaluate the idea.<br />

edu.oriented are those ideas whose initial description either lacks educational objectives or has such a list<br />

thereof, but this is inaccurate or unclear. Here is an example of the first kind:<br />

The Romantik Tour travel agency has contacted our software house to produce a CD-ROM<br />

catalogue that is to be distributed both to customers and to tour operators. Moreover, this catalogue<br />

may be later upgraded by the agency.<br />

The lack of educational orientation is pinpointed by the instructor:<br />

The idea is briefly and well expressed, with respect to functionality. But what about the objectives<br />

of your laboratory work?<br />

As an example of the second kind, here is a proposed list of objectives:<br />

feasibility study, risk management, quality objectives, production process architecture, usability<br />

and accessibility, identification of resources and operational constraints, operational plan, time and<br />

cost estimation.<br />

The instructor replies:<br />

The proposal is not clear because objectives and sub-objectives are listed with no logical structure,<br />

could you please reorganize them? Moreover, I suggest to add workgroup planning to your<br />

objectives, because the group is very large, hence it needs careful management.<br />

The advice is not limited to point out a problem in the formulation of the educational objectives, but it also<br />

contains an additonal suggestion aimed at getting the team started with proper attention to effectiveness of its<br />

own organization. This example highlights the guidance role of the instructor in operational terms, besides<br />

conceptual ones.<br />

qual.oriented are ideas that feature quality objectives, but in an unclear manner.<br />

team set-up are discussions that, besides clarifying, refining or orienting a project idea, are also exploited to<br />

define the project team.<br />

unfinished are those idea discussions which leave unanswered questions. For example, the aforementioned<br />

discussion of the “RomantikTour” idea belongs to this category, as the students never satisfied the instructor's<br />

request.<br />

59


For each column in Table 2, the sum of the discussion counts placed under the aforementioned categories need<br />

not coincide with the total number of ideas, since those categories are not mutually exclusive, but also because a<br />

few ideas (approximately 10% of the total number over the seven course editions) directly show up as started<br />

projects, without going through a discussion between students and instructors.<br />

After this due premiss, a quick look at the data in Table 2 immediately reveals that the by far most frequent<br />

reason for discussion has to do with the educational orientation, whose lack or unclarity affects almost a half of<br />

the ideas. Refinement of the functional content of the idea takes the second place, finally followed by orientation<br />

with respect to quality objectives. This one featured a rapidly growing trend until the previous course edition,<br />

which fact may be explained by the growing importance given by the instructors to quality objectives, in the<br />

course of time. The exploitment of project idea discussions as an opportunity to build up the project team<br />

exhibits a steadily growing trend throughout the whole sequence of course editions.<br />

Table 3 allows us to evaluate other interesting data about the project start-up discussions, expressed as average<br />

values. The first two columns respectively concern the size of the project team and the number of notes posted to<br />

those discussions. Because these usually consist of a sequence of questions and answers between instructors and<br />

students, we put forward this value as another quantitative indicator of the relevance of the instructors’ role.<br />

Table 3. Statistical averages<br />

edition team size posted notes start week start week start week start week<br />

refined edu.oriented qual.oriented<br />

1999-2000 4.75 1.20 5.67 — — —<br />

2000-2001 3.45 3.43 1.86 1.50 1.43 —<br />

2001-2002 3.43 3.92 11.37 4.50 7.50 2.25<br />

2002-2003 3.17 3.88 17.18 11.13 13.71 13.67<br />

2003-2004 3.53 4.53 11.41 17.29 9.17 10.57<br />

2004-2005 4.07 4.32 13.60 10.14 7.41 10.65<br />

2005-<strong>2006</strong> 3.44 5.97 8.29 7.38 6.47 8.17<br />

Starting from the second course edition, for each idea discussion the smallest number of notes is 2, as it happens<br />

when the idea is immediately judged to be correctly formulated, whereas more notes are to be expected when<br />

problems affect its first formulation. In the first course edition the average value of the number of posted notes<br />

falls below 2 because, unlike the subsequent course editions, all project idea proposals were posted to a single<br />

forum, where they most often did not receive any feedback from the instructor—that was rather given<br />

personally, during project discussion time in the lab. Several other changes have occurred in the course<br />

organization over the years, testifying to the fact that this has been a learning experience for the instructors no<br />

less than for the students.<br />

The three rightmost columns provide one with a few indicators on how the various start-up discussions, added in<br />

the course of time, come to build a sort of collective memory that helps newcomers to correctly formulate their<br />

own idea. The term “start week” denotes the number of weeks spent before proposers of a project idea introduce<br />

their idea; this number is counted starting from the day when the instructors give a classroom introduction to<br />

how should team projects be set up.<br />

The subsequent three columns report the same statistic, but only computed over the project ideas which fall<br />

under one of the three aforementioned categories which have been shown to be the main reasons of discussions.<br />

Except for one case (precisely refined ideas at the fifth course edition), the averages in these three columns are<br />

lower than the general average; this fact clearly tells that ideas submitted later are more frequently found correct,<br />

probably because proposers have learnt from the experience made by their preceding, eager-to-start colleagues.<br />

It seems that no similar improvement can be ascribed to progress throughout subsequent course editions, for<br />

reasons that are as yet to be fully understood. However, there appears to be a significant correlation between the<br />

rate of correctness of initial formulation of ideas and average start week throughout different course editions; if<br />

c y denotes the percentual number of correct ideas at course edition y, while w y denotes the average start week at<br />

the same course edition, our data yield a value of 0.805 for the correlation coefficient between these two<br />

statistics.<br />

60


Related work<br />

There is no standard methodology for software engineering education. The basic rationale for including student<br />

projects in the educational activities is to let students get a foretaste of what a professional in the field is<br />

expected to do, and how. Not all software engineering courses include such a laboratory part. Even those which<br />

do so, may largely differ in the way projects are set-up and organized, e.g. whether or not do they involve<br />

collaborations with industrial sponsors, and, in either case, who is in charge of defining project objectives and<br />

task assignments.<br />

Because of the rapid pace of change in software engineering methods and technologies, industrial impact<br />

through education is a persistently hot topic in specialized conferences on the subject, see for example the panel<br />

CSEET (<strong>2006</strong>) at the forthcoming edition of CSEET.<br />

The laboratory project set-up in the present case study is motivated by firm reasons of principle. The instructors<br />

held a constructivist viewpoint, which may be traced back to Vygotsky's social interaction theory Vygotsky<br />

(1978), takes due account of the educational value of training students to take up the responsibility of setting<br />

their own learning objectives, and exploits the opportunities of web-based cooperation to implement a<br />

scaffolding strategy to that take-up purpose.<br />

While the debate on software engineering education has been most often centered on the specific traits of the<br />

discipline so far, the importance of adopting proper educational theories is starting to be recognized—see for<br />

example Basili & Basili (<strong>2006</strong>) at the aforementioned conference. Our approach seems consistent with that<br />

direction.<br />

Conclusions<br />

In this paper we have presented and briefly analyzed a case study in software engineering education, spanning<br />

over a seven-year evolution, characterized by a blend of educational techniques: traditional classroom lectures,<br />

textbook and lecture notes, as well as a web-based cooperation platform, supporting interaction and selforganization<br />

of laboratory projects. The laboratory project set-up in the present case study is motivated not only<br />

by reasons of principle, as recalled in the previous section, but also by practical constraints. The largely selforganized<br />

character of the laboratory project work turns out to be a necessity, since the bare number of students<br />

enrolled in the course would make it impossible to provide a continued guidance and daily organization of all<br />

projects by just one or two instructors. In other words, scarcety of human resources puts a severe limit on the<br />

level of guidance that could be effectively provided under a different arrangement.<br />

The following conclusions can be drawn from the case study summarized in the previous sections.<br />

First, regardless of their “easiness”, the first questions proposed as exercises always turn out to be the most<br />

difficult to get answered properly. The main difficulty is to be traced, in this case, to the novelty of the<br />

methodological mindset wherein the initial subject—applicability of software engineering principles to web<br />

design case studies—is framed.<br />

Second, the issue of endowing project proposals with clearly stated educational objectives has persistently<br />

turned out to be the most significant reason for delayed set-up of team projects, despite the provision of detailed<br />

guidelines to this purpose in the laboratory edu-cational material. The reason for this difficulty is, again, the<br />

novelty of a methodological mindset where metacognition is explicitly meant to play the most significant role.<br />

Students are just not used to that, they are rather inclined to focus on the functional features of their intended<br />

product than on their own educational gain. This is surely not surprising, at the initial project set-up stage, nor is<br />

it necessarily a bad sign; it just highlights the usefulness of scaffolding strategies where the level of guidance<br />

provided by instructors is significant at the early stages, and then progressively decreasing.<br />

Third, regardless of the variety of alternative organization models proposed for group work, the vast majority of<br />

students privilege the most established one, viz. that which appears to be most popular from the previous<br />

editions of the course. This can be seen as an instance of the well-known “popularity is attractive” principle from<br />

the theory of network evolution Dorogovtsev & Mendes (2003).<br />

Our final, perhaps most interesting observation is that, even in self-organized project-based learning, the<br />

instructors' role remains critical, especially in the initial orientation provided to the students in order to grasp the<br />

61


educational goals and the various roles they may undertake to achieve them. This educational aim seems to be<br />

most effectively achieved by prompting and exploiting all available opportunities for meta-cognitive<br />

reinforcement, which fosters deeper inquiry capabilities, knowledge and insight.<br />

References<br />

Basili, V.R., & Basili, P.A. (<strong>2006</strong>). Software Engineering Instruction and Education Theory: a Dialogue,<br />

retrieved July 24, <strong>2006</strong> from http://db-itm.cba.hawaii.edu/cseet<strong>2006</strong>/Basili.php.<br />

BSCW (2005). Basic Support for Cooperative Work, retrieved <strong>October</strong> 2, <strong>2006</strong> from<br />

http://bscw.fit.fraunhofer.de/.<br />

CSEET (<strong>2006</strong>). CSEET <strong>2006</strong> Panel: Industrial Impact through Education – Lessons Learned from Barry<br />

Boehm's Contributions to Software Engineering, retrieved July 24, <strong>2006</strong> from<br />

http://www.sbl.tkk.fi/CSEET06panel.<br />

Chang, K.-E., Sung, Y.-T., & Lee, C.-L. (2003). Web-based collaborative inquiry learning. Journal of Computer<br />

Assisted Learning, 19 (1), 56-69.<br />

Dorogovtsev, S., & Mendes, J. (2003). Evolution of Networks, Oxford, England: OUP.<br />

Nesbit, J., & Winne, P. (2003). Self-regulated inquiry with networked resources. Canadian Journal of Learning<br />

and Technology, 29 (3), retrieved <strong>October</strong> 2, <strong>2006</strong> from http://www.cjlt.ca/content/vol29.3/cjlt29-3_art5.html.<br />

Piccinini, N. (2005). Norme d'uso del BSCW. Integrazione per il lavoro di laboratorio, retrieved <strong>October</strong> 27,<br />

<strong>2006</strong> from http://amarena.sci.univr.it/edu/pub/bscw.cgi/2482086/NormeUsoBSCWLab-in.html.<br />

Piccinini, N., & Scollo, G. (2005). Fare laboratorio in rete: tracce di un’esperienza di web-learning. DIPAV<br />

Quaderni, 12/13, 143-164.<br />

Salovaara, H. (2005). An exploration of students’ strategy use in inquiry-based computer-supported<br />

collaborative learning. Journal of Computer Assisted Learning, 21 (1), 39-52.<br />

Scollo, G. (2005). Architetture software e Ingegneria del software, Note delle Lezioni, retrieved <strong>October</strong> 9, <strong>2006</strong><br />

from http://amarena.sci.univr.it/edu/pub/bscw.cgi/2481306/AIS0506-in.<br />

Vygotsky, L. (1978). Mind in Society, Cambridge, MA: Harvard University Press.<br />

62


Tomsic, A., & Suthers, D. D. (<strong>2006</strong>). Discussion Tool Effects on Collaborative Learning and Social Network Structure.<br />

Educational Technology & Society, 9 (4), 63-77.<br />

Discussion Tool Effects on Collaborative Learning and Social Network<br />

Structure<br />

Astrid Tomsic and Daniel D. Suthers<br />

Department of Information and Computer Sciences, University of Hawai`i at Manoa, Honolulu, HI 96822, USA<br />

Tel: 1-808-383-0411<br />

atomsic@hawaii.edu<br />

suthers@hawaii.edu<br />

ABSTRACT<br />

This study investigated the social network structure of booking officers at the Honolulu Police Department<br />

and how the introduction of an online discussion tool affected knowledge about operation of a booking<br />

module. Baseline data provided evidence for collaboration among officers in the same district using e-mail,<br />

telephone and face-to-face media but showed minimal collaboration between officers in different districts.<br />

On average, knowledge of the booking module was low. After introduction of the online discussion tool the<br />

social network structure changed, showing an increase in collaboration between different districts and an<br />

increase in knowledge of the booking module, even though frequency of collaboration did not increase<br />

significantly. The study suggests that the formation of new collaborative ties is more significant for learning<br />

through information sharing in social networks than raw frequency of interaction. This work is framed by<br />

theories of knowledge building, social architecture, and communities of practice.<br />

Keywords<br />

Collaborative learning, Knowledge building, Social network analysis, Online discussion<br />

Introduction<br />

This paper reports on a study that investigated how the introduction of a web-based discussion board changed the<br />

social network of an organization, with concomitant changes in distribution of knowledge concerning the use of<br />

a new tool that had been recently introduced into the organization. The study was situated in the Honolulu Police<br />

Department (HPD), a multi-district and hence distributed organization. At the outset of our study, the<br />

Department had recently introduced a client-server Records Management System (RMS), replacing a centralized<br />

system requiring that paper-based forms be submitted from district sites. The first phase of the study documented<br />

the social network structure between six different physical locations and distribution of officers’ knowledge of<br />

the RMS. The second phase of the study introduced an online discussion tool into a subset of the districts that<br />

allowed officers to collaborate with their colleagues in the participating districts. Social Network Analysis<br />

(SNA) indicated that the introduction of the new software led to a change in the social network between districts.<br />

The results of a second knowledge assessment showed that there was also an increase in knowledge among<br />

booking officers using the online discussion tool. Social network structure, specifically information sharing<br />

across distributed locations, appeared to be more important than frequency of interaction. The study contributes<br />

to our understanding of the value of computer-mediated communication in distributed organizational contexts<br />

where other forms of communication are available and continue to be used.<br />

The paper begins with an introduction to relevant aspects of the organization and practice of the Honolulu Police<br />

Department and the tool that mandated a change in practice. The next section provides theoretical background,<br />

focusing on social architecture, knowledge building and communities of practice perspectives on learning in<br />

organizations, and discusses the potential benefits of computer mediated communication for encouraging<br />

collaboration among participants and supporting communities of practice. The remainder of the paper reports on<br />

the methodology, results and implications of the study itself.<br />

A Change of Practice in the Honolulu Police Department<br />

The Honolulu Police Department (HPD) is responsible for police operations in the City and County of Honolulu,<br />

which consists of the entire island of O`ahu. Units included four districts and two support units. This study<br />

focused on booking officers, police officers who have the primary duty of processing arrestees as they are<br />

brought into a district police station. Depending on the severity of the crime or availability of holding cells, the<br />

booking officer may decide to transfer the arrestee to the main booking station, known as the Central Receiving<br />

Division (CRD).<br />

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the<br />

copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies<br />

are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by<br />

others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior<br />

specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.<br />

63


A new computerized Records Management System (RMS) was introduced by the Information Technology<br />

Division (ITD) in August 2003. Prior to the implementation of the RMS, a standalone DOS-based booking<br />

program was used to process and track arrestees. This booking program was only available in CRD. Officers at<br />

other districts filled out paper-based forms and sent them to CRD for processing. The new RMS was clientserver<br />

based and contained an integrated booking module that made entering and processing of bookings<br />

possible from any district on the island of O`ahu. With the new RMS came the need to train officers on how to<br />

use the RMS to enter and search for information. Entering accurate data and being capable of searching criminal<br />

history records is a mandatory and vital part of a police officers’ daily duty and essential to effective police<br />

work. The degree of knowledge that an officer has about how to use the RMS system to locate pertinent<br />

information has a direct bearing on the outcome of criminal cases.<br />

At the time that the RMS was introduced, officers used a combination of e-mail, telephone, face-to-face<br />

discussions and other means to collaborate with each other in order to locate information, raise issues and<br />

propose more efficient ways to use the RMS. The relevant expertise was distributed throughout the organization:<br />

the ITD staff had a good understanding of how to use the RMS in more general terms (e.g., search for a name),<br />

while police officers had a better understanding of what information they needed to find and how it related to<br />

business processes, but not necessarily the best way to find it. The existing collaboration process between police<br />

officers and the ITD staff was inefficient because knowledge was spread over many types of media, and due to<br />

the nature of these media needed to be repeated frequently to different parties at different times, was often lost<br />

and could not be easily located and improved upon. Coupled with this, officers did not have a clear<br />

understanding of who knew what and tended to address all information requests to a single ITD staff member.<br />

This situation presented us with an opportunity to investigate how computer mediated collaboration (CMC),<br />

specifically an asynchronous discussion tool, might improve knowledge sharing and knowledge building within<br />

this distributed community of practice. Our intervention will be described in a subsequent section, but first we<br />

consider theoretical motivations.<br />

Learning in Organizations<br />

An intervention intended to influence learning in an organization must consider how such learning might take<br />

place. Although various social learning theories are available, the most relevant theories for the present study are<br />

those that emphasize the community aspects of learning. We draw particularly upon knowledge building theory<br />

(Scardamalia & Bereiter, 1996), Bogenrieder’s (2002) sociocognitive theory, and communities of practice theory<br />

(Wenger, 1998; Wenger, McDermott & Snyder, 2002).<br />

Knowledge Building<br />

Knowledge building (Scardamalia & Bereiter, 1996) refers to the intentional pursuit of advances in collective<br />

understanding undertaken by a community. According to Scardamalia & Bereiter (1996), it is “work on the<br />

creation and improvement of ideas. The dynamic is social, resulting in the creation of public knowledge …<br />

public knowledge can itself become an object of inquiry and the basis for further knowledge building.” At the<br />

time of our study, departmental officers and staff were engaged in negotiating practices for use of the new<br />

booking management system. The CMC technology that we introduced made statements of knowledge needs<br />

and proposed solutions public in a persistent medium available to many other community members, and hence<br />

offered the conditions for treating knowledge as an object of inquiry in this community.<br />

Extensive research motivated by knowledge building theory has been conducted in settings for formal education<br />

(schools and colleges). For example, Sha & Van Aalst (2003) conducted a study using server log data to explore<br />

knowledge building in the classroom. The techniques of Social Network Analysis (SNA, Wasserman & Faust,<br />

1995) were used to analyze student participation and interactivity in an online discussion database called<br />

Knowledge Forum. SNA was used to map the structure of relationships, the major lines of communication and<br />

patterns of interaction within the social network. It provided valuable insight into the effectiveness of the class’s<br />

ongoing efforts to improve their knowledge building and the conditions under which knowledge building<br />

occurred. The Sha & Van Aalst study sought to extend the framework for knowledge assessment. Despite a<br />

widespread emphasis on social constructivism, assessment is still based on the individual student’s final<br />

knowledge. Their study explored how server log data could be used to inform continued knowledge building. It<br />

suggested that improving connectivity within a community might provide better support for knowledge building.<br />

64


Similarly, our study used server log data to map the social network and examine the content of the posts for<br />

different behaviors. It also explored how changes in the social network affected learning.<br />

Social Architecture<br />

Bogenrieder (2002) argues that organizational learning is a combination of both social-relational and cognitive<br />

activity. According to Bogenrieder, the social-relational aspect consists of the social networks fostered by the<br />

organization while socio-cognitive conflict (Doise & Mugny, 1984) is the vehicle for nurturing cognitive<br />

activity. Socio-cognitive conflict has two conditions that must be met for learning to take place. The first<br />

condition is that a social relationship exists between the participants and the second condition is cognitive<br />

diversity. When participants contribute different ideas, these differences trigger learning through socio-cognitive<br />

conflict (and, we would argue, other mechanisms of intersubjective meaning making: see Suthers, <strong>2006</strong>). For<br />

example, in a police department officers have different educational backgrounds and levels of experience in<br />

police work, and varied expertise in certain types of crimes. If sharing of ideas between officers is encouraged,<br />

these differences can be leveraged to produce more effective practices for crime reduction.<br />

Bogenrieder (2002) suggests that social architecture can be used as an instrument to build relationships for<br />

learning. As previously discussed, both cognitive diversity and contact between diverse people are necessary for<br />

organizational learning. The specific design for this contact—the social network—depends on the goal<br />

uncertainty and technical uncertainty that characterize the problem situation. Goal uncertainty is “ambiguity<br />

about the preferences or goals the decision-maker aims to satisfy” and technical uncertainty is “uncertainty in<br />

parameters, input data and initial states (resulting from 'inexactness' and 'conflicting evidence')” (Weber, 2000).<br />

Where conditions of high goal and technical certainty exist, instrumental network structures are satisfying.<br />

However, where goal certainty coexists with technical uncertainty, an effective network structure is sought based<br />

on the distribution of knowledge. In the case of officers learning the new RMS, there is high goal certainty<br />

(officers know what they want to do), but high technical uncertainty (officers don’t necessarily know how to do<br />

it). In these types of networks, the network paths are not pre-determined: the structure of the network forms<br />

based on participants’ seeking of the knowledge needed to resolve the technical uncertainty.<br />

Laboratory studies have indicated that some types of network structures are more effective than others for<br />

diffusing information throughout a group (Cummings & Cross, 2003). However, the task is usually pre-defined<br />

by the researcher who establishes paths for the diffusion of information. In organizations, information flow<br />

depends on the expertise of the group, as discussed above. Therefore, while certain network structures may be<br />

more efficient for information diffusion in an experiment, these structures may not be effective for leveraging<br />

group expertise in an organization. The literature suggests that from a cognitive perspective, network structures<br />

with greater integration (connectivity) may be more effective for leveraging group expertise. Studies of<br />

transactive memory suggest that groups benefit from knowledge of who knows what in the group (Liang et. al.,<br />

1995, Moreland & Myaskovsky, 2000). Work in diffusion of innovations suggests that hierarchical network<br />

structures are both inefficient and result in the degradation of information quality (Rogers, 1995; Valente, 1995).<br />

Communities of Practice<br />

The theory of Communities of Practice (CoP; Wenger, 1998; Wenger et al., 2002) provides further conceptual<br />

tools for understanding how the collective knowledge of a community is sustained and adopted to new<br />

situations. According to this theory, a CoP consists of people mutually engaged in ongoing participation,<br />

oriented towards a common domain that is addressed through a shared repertoire: a “group of people who share a<br />

concern, a set of problems, or a passion about a topic, and who deepen their knowledge and expertise in this area<br />

by interacting on an ongoing basis” (Wenger, et al. 2002). When members of a CoP are distributed throughout an<br />

organization in different teams, the interaction between them required to sustain knowledge is at risk.<br />

Information technology can be valuable for bringing community members together to share and resolve<br />

problems, as discussed in the next section.<br />

According to CoP theory (Wenger, 1998), the knowledge of an organization—the repertoire of its communities<br />

of practice, such as the police officers—is sustained through the dynamic interplay of the duality of participation<br />

and reification. A duality is not a pair of polar opposites, but rather a dynamic productive pair: increasing one<br />

may well require increasing rather than decreasing the other. When discussing communities of practice,<br />

participation refers to how we engage together in an activity, and through that engagement generate, affirm and<br />

replicate patterns of engagement: our practices. Reification refers to ways in which these practices become<br />

65


eflected in artifacts and other structures that help replicate the practices. For example, the practices of a police<br />

department are reflected in the structure of its booking forms. This structure supports replication of those<br />

practices in the moment by reminding participants of what they need to do, and in the long term by serving as a<br />

guide through which new members of a CoP can be brought into its practice. Reifications of practice are not<br />

restricted to physical artifacts and information technologies. Terminologies, ways of talking, conventions of<br />

social greeting, etc. can also serve as reifications of practice. Importantly, participation that engages a reification<br />

is not just controlled and constrained by that reification, but also reaffirms and sustains the meaning of the<br />

reification, and can change the meaning of (reinterpret) the reification. Participation in an online discussion<br />

results in reifications in the form of messages that can guide future participation or be reinterpreted through that<br />

participation.<br />

Computer Supported Social Networks as Socio-Technical Capital<br />

Wellman (1996) claims that Computer Supported Social Networks (CSSN’s) support a focus on information<br />

exchanges. “People can easily post a question or comment and receive information in return. Broadcasting<br />

queries through CSSN’s increases the chances of finding information quickly and alters the distribution patterns<br />

of information. It gives those working in small or distant sites better access to experienced, skilled people.”<br />

Additionally, “online information flows spill over unexpectedly through message forwarding, providing access<br />

to more people and new social circles, thus increasing the probability of finding those who can solve problems”<br />

(Kraut & Attewell, 1993). For example, officers don't know who to ask for information, but a discussion board<br />

works to their advantage because it doesn't require that officers know whom to ask: they need only post a<br />

message for all to read. Computer mediated communication “turns communication into substance”, a reification<br />

that can be accessed and elaborated on by others at different times and places in a manner not possible with<br />

volatile media such as speech. Furthermore, the value of CSSNs is not limited to sharing information. The social<br />

relationships supported by CSSN technology in conjunction with the affordances of that technology for<br />

continuing those relationships in certain ways collective constitute socio-technical capital: “a resource that can<br />

be accumulated and whose availability allows people to create value for themselves or others” (Resnick, 2002).<br />

As soon as an officer receives a reply, not only has new information become available, but also his or her social<br />

network has expanded. If the reply is helpful, the officer now knows who to ask for further information on that<br />

particular topic, as do others who are observing the interaction, and obligations to be helpful in a reciprocal<br />

manner are created. For these reasons, we expected that it would be beneficial for police officers and CRD and<br />

ITD staff to collaborate in an online environment. This expectation was assessed by (1) studying the changes in<br />

the network structure, (2) investigating the frequency and degree of collaboration, and (3) analyzing the content<br />

of the posted messages.<br />

Synthesis<br />

Organizational learning is required when practices must change to reflect changing conditions, for example, the<br />

introduction of a new tool such as the new booking system. Ideally the members of the organization engage in<br />

knowledge building, which in CoP terms can be understood as the deliberate pursuit of extensions to the<br />

collective repertoire to adapt to the new conditions. The role of a CSSN in this process is twofold, as suggested<br />

by the participation/reification duality. From the perspective of participation, a CSSN extends mutual<br />

engagement to a larger collection of people, in this case distributed across multiple sites. In an expanded<br />

network, cognitive diversity increases and innovations are shared more widely, as the reifications resulting from<br />

participation are shared more widely. From the perspective of reification, the persistent reifications generated by<br />

participation in a CSSN can influence practice at other times and places, even the practice of those not involved<br />

in the production of those reifications, and can also be reinterpreted by others. Thus, for reasons influencing both<br />

participation and reification, we expected the introduction of an online discussion tool to improve the scope and<br />

extent of mutual engagement in the knowledge building process of developing new practice around the new<br />

booking system.<br />

Study Design and Methods<br />

The foregoing theoretical discussion concluded that a CSSN can increase knowledge building in an organization<br />

by enabling and expanding interaction between a greater diversity of community members across space and<br />

time, through networking and persistent representations, respectively. One way to determine whether this<br />

happens in practice is to introduce a CSSN into an organization, observe whether that introduction expands the<br />

66


social network of the organization (whether there is more interaction between more people), and observe<br />

whether this change is responsible for advances in knowledge (those who interact increase their knowledge<br />

concerning the topic about which they interact). Our study design applies this strategy to address the following<br />

research questions.<br />

Research Questions and Hypotheses<br />

Research Question I. How will the introduction of an online discussion tool affect the existing social network<br />

structure at the Honolulu Police Department for officers learning the booking module for a new Records<br />

Management System (RMS)?<br />

Hypothesis 1 (H1). There will be an increase in collaboration about the booking module of the RMS between<br />

participants in different districts using the online discussion tool compared to participants who do not, using pre<br />

and post interview instruments and social network analysis (SNA). Rationale: CMC expands the potential social<br />

network over space and time, and officers will take advantage of this expanded network to seek the knowledge<br />

needed to reduce technical uncertainty.<br />

Hypothesis 2 (H2). There will be an increase in frequency of collaboration about the booking module of the<br />

RMS between participants using the online discussion tool compared to participants who do not, using pre and<br />

post interview instruments and SNA. Rationale: CMC, being asynchronous and persistent, makes collaboration<br />

easier, and the need for knowledge building makes collaboration desirable.<br />

Research Question II. How will use of the online discussion tool influence knowledge of the RMS booking<br />

module?<br />

Hypothesis 3 (H3). There will be an increase in knowledge of the booking module of the RMS for participants<br />

using the online discussion tool compared to participants who do not, using pre and post survey instruments.<br />

Rationale: A CSSN enables sharing of information and encourages knowledge building by increasing cognitive<br />

diversity and supporting reifications that make knowledge claims open to inspection and interpretation.<br />

Participants<br />

Booking officers were randomly selected from a pool of officers of different ages with different cultural and<br />

academic backgrounds from four (4) different locations (districts) on O`ahu. Twenty (20) officers were selected<br />

from districts 5 and 8; of these, fifteen (15) chose to participate. We call these officers Group A. Another twenty<br />

(20) officers were selected from districts 2 and 3; of these, sixteen (16) chose to participate. We call these<br />

officers Group B. Along with these thirty-one (31) police officers, four (4) CRD civilian booking staff and seven<br />

(7) ITD staff participated in the study. Each district collaborated with the ITD staff and CRD staff. The districts<br />

were at different physical locations than each other and the ITD and CRD staff. This physical separation enabled<br />

us to minimize cross-collaboration between the two districts that were provided with access to an online<br />

discussion tool (Group A) and those that were not (Group B).<br />

Intervention<br />

Our intervention consisted of the introduction of an online discussion board dedicated to discussion of the use of<br />

the RMS. This discussion board was made available to Group A and the CRD and ITD staff, but not to Group B,<br />

enabling the foregoing hypotheses to be tested in a quasi-experimental design.<br />

An online discussion board is a viable tool to support knowledge building in a distributed community of practice<br />

because it can support spatially and temporally distributed interaction and it offers persistent representations<br />

through which public knowledge becomes an object of inquiry. The HPD officers are distributed across physical<br />

HPD districts, yet are faced with a similar set of issues that non-booking officers may not be familiar with. They<br />

use the same software module in the RMS and are required to be familiar with specific procedures for booking.<br />

They must also be familiar with basic troubleshooting procedures because the software support staff (ITD) are<br />

separated from the officers who must put the knowledge to use. An online discussion board offers a shared and<br />

persistent forum for booking officers to increase their awareness of system-wide issues and booking practices.<br />

Other media such as face-to-face, e-mail and telephone lack persistence and are limiting for officers working in<br />

different districts or different shifts. However, a discussion board is not without its limitations: the challenges for<br />

67


any distributed community include building trust and maintaining online participation (Preece 2000; Wenger et<br />

al., 2002).<br />

The particular online discussion tool used in this study was Discus, available at www.discusware.org. The<br />

authors are not associated with the Discus project. The research reported in this paper is concerned with the<br />

potential impact of the genre of web tools represented by Discus, not with evaluation of Discus’ specific design<br />

features. The first author chose this tool based on the following features. It used a standard and familiar threaded<br />

format (see Figure 1). Being web-based, Discus was accessible from every computer that ran the booking<br />

module for the Records Management System. Questions could be posted as soon as an issue occurred, and an<br />

officer could access information anytime, not only when the ITD staff or CRD staff were available. Discus also<br />

offered the ability for officers to see who posted a message. This is important in community building because if<br />

an officer finds a particular post helpful, they can continue asking the author of that post for further information.<br />

It also allows officers to become familiar with who knows what about the RMS and thus who to ask for<br />

clarification or further explanation.<br />

Procedure and Task<br />

Figure 1. Layout of the threaded discussion format in the Discus tool<br />

The study was conducted over a six-week period in March/April 2004. (At the end of this period, a departmentwide<br />

reassignment of officers changed the composition of the districts.) Participants were asked to complete two<br />

(2) surveys (pre and post) and two (2) interviews (pre and post). The interview layout and content was adapted<br />

from surveys created by Krackhardt & Haythornthwaite (1998). Both the surveys and interviews were piloted<br />

with two (2) officers who did not participate in the final study and was revised according to the officers’<br />

feedback and suggestions for improvement.<br />

The pre-survey was used to determine prior knowledge of the booking module of the RMS system. It consisted<br />

of five (5) questions with an estimated duration of ten (10) minutes. Each question consisted of typical search<br />

tasks related to the booking module in the RMS that an officer is required to perform on a daily basis. For<br />

example, “What is the report number for the FRAUD incident that occurred in sector 1 beat 150 on 10/19/2003?”<br />

A pre-interview was conducted to determine the current social structure. It consisted of questions to determine<br />

with whom the participant collaborated about the booking module of the RMS during the prior three (3) weeks.<br />

It asked how well they knew this person, the frequency of these collaborations, and the media they used to<br />

collaborate. In order to benchmark the initial social network structure, data on each of the different types of<br />

communication media used was collected. This was necessary to determine what connections for knowledge<br />

building existed beforehand and whether new connections developed as a result of the introduction of the online<br />

discussion tool.<br />

Participants in Group A (districts 5 and 8), the ITD staff and CRD staff were asked to contribute to an online<br />

discussion tool. They were given introductory training on the online discussion tool and provided with a login<br />

and password to record questions, comments, ideas and suggestions about the booking module. Posting activity<br />

on the online discussion tool was initially slow, so the ITD staff posted what it thought to be useful tips on how<br />

to use the booking module in order to generate activity. The ITD staff checked the board on a regular basis to<br />

ensure that questions were answered in a timely manner. This was done to encourage participants from other<br />

68


districts to post questions. Group B (districts 2 and 3) continued to collaborate with the ITD staff, CRD staff and<br />

each other using the traditional face-to-face, e-mail and telephone methods of collaboration.<br />

A post-survey consisting of five (5) questions with an estimated duration of ten (10) minutes was given to Group<br />

A and Group B to determine whether any changes in knowledge had occurred. Similar to the pre-survey, the<br />

post-survey contained search questions on the booking module in the RMS but did not have the same questions<br />

as the pre-survey to avoid participants learning from the pre-survey. For example, “Who was the arresting officer<br />

for the arrest with report number XXXXXX?”<br />

A post-interview was conducted with the same questions as the pre-interview to determine whether any changes<br />

had occurred in the social structure between the districts.<br />

Measures and Analysis<br />

Scardamalia (2000) developed twelve descriptors that represent the “best practices” of knowledge building.<br />

Chan, Lee & Van Aalst (2001) used a modified subset to organize course evaluation and to scaffold knowledgebuilding<br />

advances. These four principles are: 1. Working at the cutting edge, 2. Progressive problem solving, 3.<br />

Collaborative effort, and 4. Identifying high points in the discourse. Sha & Van Aalst (2003) conducted a study<br />

where they focused on the pedagogical knowledge principal of collaborative effort. They analyzed the<br />

characteristics of student’s social interaction by measuring participation (posts), reciprocity (replies),<br />

connectivity (linked notes), social position (activity in the social network) and social interaction (reads).<br />

Like Sha & Van Aalst (2003), the present study measures characteristics of individual social interaction relating<br />

to collaborative effort, but groups these measurements into broader factors that investigate social interaction<br />

between districts. These factors correspond to the three hypotheses:<br />

Collaboration – defined as any new social ties between participants who had not previously<br />

communicated. This study is particularly interested in collaboration between districts. It is measured by<br />

the messages contributed to the online discussion tool and the interactions in other media reported in the<br />

pre and post interviews. This includes messages posted, replied to and read.<br />

Collaboration Frequency – defined as the number of interactions between participants in the social<br />

network. It is measured by the number of messages contributed to the online discussion tool and the<br />

number of interactions in other media reported in the pre and post interviews. This includes the number<br />

of messages posted, replied to and read.<br />

Knowledge – defined as the understanding, familiarity and proficiency of a participant with the booking<br />

module of the RMS. It is measured by the results of the search tasks in the post-survey and the presurvey.<br />

Knowledge is rated on a scale of 0 for the lowest score with no tasks correct to 5 for the highest<br />

score for all tasks correct.<br />

SAS statistical software was used to tabulate and analyze the results of the knowledge scores. The UCINet suite<br />

of programs (Borgatti, Everitt & Freedman, 1996) was used to create the data sets for SNA and a visual<br />

depiction of the socio-centric social network in the form of social network analysis diagrams. Each participant is<br />

represented as a circle or node in the network with their interactions displayed as lines. These lines have<br />

arrowheads to show the direction of each collaborative interaction. The thickness of each line represents the<br />

number of interactions or tie-strength between each node. The minimum tie-strength is zero (0) and the<br />

maximum is ten (10). However, for the combined medium diagrams, the minimum tie-strength is zero (0) and<br />

the maximum is forty (40). The district that each node belongs to is shown by the color of the node. The score<br />

that each participant receives for the collaborative learning tasks is shown by the size of each node. The<br />

minimum score for no tasks correct is four (4) and the maximum score for all correct is nine (9). The minimum<br />

of four (4) was chosen instead of zero (0) so that each node is visible on the diagram. Three SNA diagrams will<br />

be presented in the next section.<br />

A content analysis was conducted to gain a more in-depth understanding of what types of interactions were<br />

occurring on the online discussion tool. Each message posted on the discussion board was identified and<br />

categorized using a list of behaviors suggested by Rubin & Goldberg (1992). They include basic communication<br />

relations such as information seeking (IS) and information providing (IP). Other behaviors that Rubin &<br />

Goldberg (1992) term “contractual relations” are used to indicate messages where one participant posts a<br />

69


message in order to obtain a response. These are coded as confirming action (CA), seeking consensus (SC),<br />

statement of problem (SP), and statement of solution (SS). Other behaviors that showed an outcome from the<br />

discussion were coded as making a decision (MD), notifying the occurrence of an event (NE) and volunteering<br />

assistance (VA).<br />

Results<br />

Table 1 shows the knowledge scores for each district for the pre and post surveys. The two districts (2 and 3) that<br />

did not use the online discussion tool received a lower score and the same score respectively on the post survey.<br />

Three out of the four districts (8, ITD and CRD) that used the online discussion tool received higher scores on<br />

the post-survey. There was a significant effect for knowledge, t(40) = 5.41, p < .05, with discus users receiving<br />

higher scores than non-discus users. (In this and subsequent t-tests, we verified that variance probability for<br />

equal variance applies; Kerlinger & Lee, 2000).<br />

Table 1. Knowledge scores by district (percent correct)<br />

Control group Discus users<br />

Officer Group B Officer Group A Staff<br />

2 3 5 8 ITD CRD Aggregate<br />

n = 10 n = 6 n = 10 n = 5 n = 7 n = 4 n = 42<br />

Pre-survey 60% 30% 64% 53% 42.8% 60% 51.6%<br />

Postsurvey<br />

38% 30% 55% 88% 48.5% 100% 59.9%<br />

% change -22% 0% -9% 35% 5.7% 40% 8.3%<br />

Table 2 shows the types of behaviors examined for content analysis. Table 3 shows the results for the content<br />

analysis of the messages posted in the discussion tool. The most common behaviors were those of IS<br />

(Information Seeking) and IP (Information Providing). The next most common actions were RA (Requesting<br />

Action) and CA (Confirming Action). During content analysis, it was observed that different districts sometimes<br />

disagreed on and discussed their booking procedures. Various formatting issues were negotiated, such as<br />

whether to include the dashes in the social security number, how to format the state adult booking number, and<br />

what additional fields to include in the booking. Officers were used to the prior booking procedure where<br />

booking reports were typed onto pre-printed paper reports. Previously formatting was not an issue because a<br />

limited number of people entered the data from the paper reports into the booking system.<br />

Table 2. Types of content analysis behaviors<br />

Types of behavior<br />

Key to codes Key to codes<br />

IP Information providing MD Making a decision<br />

IS Information seeking VA Volunteering assistance<br />

RA Requesting action RF Raising funds<br />

CA Confirming action SF Seeking funds<br />

SC Seeking consensus PF Providing funds<br />

SP Statement of problem OP Other people<br />

SS Statement of solution H Humor<br />

NE Notifying occurrence of event<br />

Table 3. Breakdown of behaviors by district<br />

Behavior 2 3 5 8 ITD CRD Total per behavior<br />

IP 5 2 10 2 19<br />

IS 8 3 7 1 19<br />

RA 2 2 2 4 10<br />

CA 6 2 6 14<br />

RF 0<br />

SF 0<br />

PF 0<br />

SP 5 5<br />

SS 4 1 5<br />

70


SC 1 3 4<br />

MD 1 1 1 3<br />

VA 0<br />

OP 0<br />

NE 1 5 6<br />

H 0<br />

Total per district 0 0 22 12 43 8 85<br />

Table 4 shows the number of posts, replies and reads in the discussion tool by district from the server logs. ITD<br />

was the most active in posting and replying to posts. All of the districts that used the discussion tool (5, 8, ITD<br />

and CRD) were more active in reading messages posted by others compared to posting. The ratio of reads to<br />

posts was almost 41:1 in the case of ITD, 50:1 for district 5 and 22:1 for district 8.<br />

Table 4. <strong>Number</strong> of posts, replies and reads by district in the online discussion tool<br />

2 3 5 8 ITD CRD Total<br />

Posts 0 0 10 4 19 5 38<br />

Replies 0 0 13 7 23 4 47<br />

Reads 0 0 502 156 791 62 1511<br />

Figure 2 shows the initial social network with all media combined, and Figure 3 shows the social network after<br />

the introduction of the online discussion tool with all media combined. These two diagrams allow comparison of<br />

differences in the social network. The initial social network shows collaboration within each district but a lack of<br />

collaboration between districts. The post social network shows new collaboration between the districts (5 and 8)<br />

using the discussion tool. There was a significant effect for inter-district collaboration, t(40) = 2.57, p < .05, with<br />

increased collaboration across districts following the introduction of the tool. However, there was no significant<br />

effect for collaboration frequency, t(40) = -0.02, p > .05, with discus users receiving similar scores to non-discus<br />

users. Therefore the null hypothesis for H2 cannot be rejected.<br />

Figure 2. Social network before introduction of the online discussion tool. In this and subsequent figures, **<br />

indicates districts that used discus<br />

71


Figure 3. Social network after introduction of the online discussion tool<br />

Figure 4. Social network of interactions using the online discussion tool<br />

Figure 4 shows the collaborative interactions between the districts using the online discussion tool (connections<br />

via other media are omitted from this diagram). Figure 4 shows that there was new collaboration using the online<br />

72


discussion tool between district 5 and district 8 and that the overall knowledge scores (represented as size of the<br />

nodes in the graph) had increased.<br />

Officer Comments<br />

Comments from officers indicated that Discus was particularly useful because it was available no matter what<br />

booking computer they were assigned to. One user commented that the ability to post attachments such as<br />

screenshots saved time by allowing other officers to refer to the screenshot, thereby reducing the amount of<br />

textual description needed to explain a point.<br />

Discussion<br />

The results of changes to the social network, including collaboration patterns between individual participants and<br />

different districts and collaboration frequency will be discussed.<br />

Initial Social Network and Knowledge<br />

The results of the initial social network analysis (Figure 2) indicate that officers tended to collaborate within<br />

their own districts for information and rarely collaborated between districts. Furthermore, Figure 2 shows that<br />

there appeared to be significant collaboration between the individuals in the Information Technology Division<br />

(ITD), and especially between two of the central individuals (3 and 16). There was significant collaboration<br />

between individuals in the ITD and individuals in the Central Receiving Division (CRD). Each of the districts<br />

appeared to have a unique network structure, with the one commonality being a central or liaison individual who<br />

collaborated with others outside the district. This central individual was not necessarily the individual who has<br />

the highest knowledge score (34, 15, 23). Similarly, some of the individuals (10, 42, 20) who are isolated<br />

received the highest knowledge scores, indicating that these individuals are underutilized for their expertise. The<br />

initial survey showed that the average knowledge score across all districts was 51.6% (Table 1). In addition, each<br />

of the districts had a different social network structure (Figure 2). The districts with the highest scores were<br />

district 5 with 64%, district 2 at 60% and CRD at 60% (Table 1). The similarity between the network structures<br />

of these three districts is that each of the individuals collaborated with others in the same district and also with<br />

individuals in other districts. The collaboration was two-way. The network structure of the districts that received<br />

the lowest pretest scores, district 8 at 53%, ITD at 42.8% and district 3 at 30% (Table 1) tended to have one or<br />

two officers collaborate outside the district and then share the information within the district. The collaboration<br />

appeared to be mostly one-way. Interestingly, ITD staff had one of the lowest overall knowledge scores. Relating<br />

this to Figure 2, it appears that two ITD staff members (3, 16) are over-utilized, creating a bottleneck in the<br />

network between other ITD staff and other districts collaborating via these two individuals.<br />

Collaboration Between Districts (H1)<br />

The results from the social network analysis support H1, which states that there would be an increase in<br />

collaboration between participants in different districts using the online discussion tool compared to participants<br />

who did not. The results indicated that the online discussion tool encouraged new ties to be formed between<br />

participants who had not previously collaborated. There was a significant increase in collaboration between<br />

districts 5 and 8 (Figure 3) after the online discussion tool was introduced compared to the initial social network<br />

(Figure 2). The two districts that did not have access to the online discussion tool (2 and 3) showed collaboration<br />

within their own district but there was a lack of collaboration with any of the other districts (Figure 3).<br />

Collaboration Frequency (H2)<br />

H2 states that there would be an increase in collaboration frequency between participants using the online<br />

discussion tool compared to participants who did not. The results from the social network analysis did not show<br />

a significant difference in collaboration frequency between the group that used Discus and the group that did not:<br />

the null hypothesis for H2 cannot be rejected. The only significant increase in tie-strength was between<br />

participants 3 and 12. There was increased collaboration between officers in district 2 but this was mainly due to<br />

73


increased face-to-face collaboration. The level of participation in the online discussion database may explain<br />

why the frequency of collaboration between participants was low. To overcome this limitation, these results can<br />

be analyzed again once the online discussion tool has been used over a longer time period.<br />

Knowledge (H3)<br />

The results from the second survey showed an increase in average knowledge scores across all districts from<br />

51.6% to 59.9% (Table 1). Knowledge scores across districts showed that three of the four districts (district 8,<br />

CRD and ITD) that participated in the online discussion tool showed increased knowledge scores, while the two<br />

districts that were not given access to the online discussion tool showed either constant (district 3) or decreased<br />

(district 2) overall knowledge scores. The statistical test of results from the second survey supports H3: there was<br />

an increase in knowledge for participants using the online discussion tool compared to participants who did not.<br />

This difference suggests that the changes in the social network structure (Figure 3), in addition to practice over<br />

time, improved learning.<br />

Relationship Between Social Network and Knowledge<br />

The districts with the highest post-scores were CRD with the highest score at 100%, followed by district 8 at<br />

88% and district 5 at 55%. It appears that CRD received the highest score due to the increase in knowledge by<br />

two key individuals (28, 1). Staff member 28 was the only individual in CRD to contribute in the online<br />

discussion tool. The social network of district 8 changed noticeably, with increased two-way collaboration<br />

between officers 30, 14, 40 and 34 (Figure 3). These individuals were active participants in the online discussion<br />

tool, communicating with ITD (Figure 4). The social network within district 5 remained essentially the same.<br />

Officers 18, 21 and 26 increased their collaborative learning scores; however, of these only officer 26<br />

participated in the online discussion tool. It appears that officers 18 and 21 had increased two-way collaboration<br />

to other individuals (21 to 4, 11 and 18 to 12) who did participate in the online discussion tool (Figure 4). ITD<br />

marginally increased knowledge scores. This can be explained by the lack of participation by ITD staff in the<br />

online discussion tool. Only staff members 38, 3 and 16 chose to participate (Figure 4), with individual 38 being<br />

the only one with an increase in knowledge score. As part of the study design, district 2 and district 3 did not<br />

have access to the online discussion tool. District 2’s social network structure changed as a result of increased<br />

face-to-face collaboration. Officers 5, 32, 15 and 24 marginally increased their knowledge scores. This may be<br />

explained by the social connections that these officers have outside of their own district. District 3’s social<br />

network structure remained the same, as did their overall knowledge score at 30%.<br />

The results showed that there was an increase in knowledge scores for 10 out of the 14 individuals in the online<br />

discussion tool, with two of these (3 and 16) receiving the maximum scores in both pre and post surveys (Figures<br />

2 and 3). There was an increase in knowledge scores for 4 out of the 16 individuals who did not participate in the<br />

online discussion tool. These four officers (all from district 2) have social connections outside of their own<br />

district. This overall distribution of scores suggests that the gains may be attributed to the use of the online<br />

discussion tool. Even though there was some variation in the social network structure of each district over time<br />

(pre versus post social network), the overall knowledge scores indicate that those districts that used the<br />

discussion tool improved their knowledge. An appropriate example is the change in the social network structure<br />

in District 2. They did not use the online discussion tool, but despite the network changes using other media<br />

(Figure 2), their overall knowledge score did not increase.<br />

Content Analysis<br />

Results from the content analysis of the messages posted using the online discussion tool indicated that all of the<br />

districts that had access to the online discussion tool used the board mainly to seek and provide information (IS =<br />

19, IP = 19 in Table 3). District 5 displayed the most information seeking (IS) behavior. They have the most<br />

integrated network structure (Figure 3). It is interesting that all districts participated in information providing<br />

(IP), not just ITD and CRD as expected. To a lesser extent, the districts used the board to request actions (RA =<br />

10) and confirm completed actions (CA = 14). An example of this is where a participant would ask participants<br />

in other districts to type data into the booking module in a certain way (see informal observations concerning the<br />

negotiation of formatting, previous section).<br />

74


Participation by Reading<br />

The logs indicate that many of the participants chose not to post messages but were active in their reading of<br />

messages posted by others. The ratio of reads to posts was almost 41:1 in the case of ITD, 50:1 for district 5 and<br />

22:1 for district 8. This indicates that the frequency of collaboration by posting and replying to messages was<br />

low, with most participants preferring to only read messages. Analysis of the knowledge results indicates that<br />

despite a low level of posting, the participants who only read the posts of others increased their own knowledge<br />

scores. Wenger et al. (2002) suggest that the learning of “lurkers” may be of significant advantage in some<br />

online communities. Although not included in the original goals of this study, further analysis of the behavior<br />

and learning of lurkers in a social network should be conducted.<br />

Conclusions and Future Work<br />

Findings from this study have indicated that the introduction of an online discussion tool had significant effects<br />

on the social network and learning of officers working with a booking module of the RMS at the Honolulu<br />

Police Department. Results supported the hypothesis that there would be an increase in collaboration between<br />

participants in different districts in which the online discussion tool was available compared to participants in<br />

districts where the tool was not available. The results did not support the hypothesis that there would be an<br />

increase in collaboration frequency between participants. However, results supported the hypothesis that there<br />

would be an increase in knowledge for participants who used the online discussion tool compared to participants<br />

who did not use the tool. These results along with the lack of knowledge gain in a district that increased its<br />

internal social networking without CMC (the exceptions are individuals who had outside contacts) suggest that<br />

increased interaction within a group has less influence on knowledge gains than the expanded size of a social<br />

network enabled by CMC. This finding might be explained by the greater cognitive diversity of a larger network.<br />

The nature of the media must also be considered: while a quantitative social network analysis counts contact<br />

through volatile media (such as face to face speech or telephone) the same as contact through persistent media<br />

(such as a discussion board), persistent media can reach more people over time and support reflection on the<br />

contributions. Content analysis and the finding that lurkers achieved knowledge gains also suggest that the<br />

discussion tool supported information sharing and coordination of action over time and space. In general, CMC’s<br />

value to an organization should not be judged merely based on whether interaction becomes more frequent:<br />

benefits also lie in increased connectivity and persistence of information.<br />

This study has shown that it is possible to change the social network structure from one that presents barriers to<br />

knowledge sharing to one that promotes learning. However, it is clear that introduction of a collaboration tool is<br />

not sufficient by itself. For example, it is necessary to maintain activity in the online discussion tool in order to<br />

create value that encourages further participation, which in turn creates further value. Participants need support<br />

in learning how to use the new collaboration technology, and how to adapt it to work processes and social<br />

processes. Technology supported media can be used to enable collaboration between participants who would not<br />

necessarily collaborate through any other type of media, but the CMC does not operate in isolation. Various<br />

media support collaboration within the social network, and synergy with communication in other media may<br />

amplify the impact of the CMC tool.<br />

This study was conducted over a six-week period due to time constraints imposed in part by movement of the<br />

participating police officers. It would be useful to evaluate the results of a similar study held over a longer period<br />

of time. Further research could examine other independent variables, such as the introduction of different media<br />

types, or dependent variables, such as how the nature of the interaction changes between participants in the<br />

social network. Studies conducted in different types of organizations and environments and examining how<br />

different media are used for different work and social process would be of value. In general, the objective is an<br />

empirically grounded theory that can predict how the introduction of different media types and practices<br />

surrounding their use would affect social networks and their implications for knowledge building within a<br />

community of practice.<br />

Acknowledgements<br />

The authors gratefully acknowledge the cooperation of the officers and staff of the Honolulu Police Department.<br />

The second author was supported by the National Science Foundation under award #0093505.<br />

75


References<br />

Bogenrieder, I. (2002). Social architecture as a prerequisite for organizational learning. Management Learning,<br />

33 (2), 192-212.<br />

Borgatti, S., Everitt, M. & Freeman, L. (1996). UCINET IV Version 1.64. Natick, MA: Analytic Technologies.<br />

Chan, C., Lee, E. & Van Aalst, J. (2001). Accessing and fostering knowledge building inquiry and discourse.<br />

Presented at the 9th Biennial Meeting of the European Association for Learning and Instruction, Switzerland.<br />

Cummings, J. N. & Cross, R. (2003). Structural properties of work groups and their consequences for<br />

performance. Social Networks, 25, 197-210.<br />

Doise, W. & Mugny, G. (1984). The Social Development of the Intellect. Oxford: Pergamon Press.<br />

Kerlinger, F. & Lee, H. (2000). Foundations of Behavioral Research. 4th ed. Harcourt College Publishers.<br />

Krackhardt, D. & Haythornthwaite, C. (1998). A social network study of the growth of community among<br />

distance learners. Information Research, 4 (1). Available at: http://informationr.net/ir/4-1/paper49.html.<br />

Kraut, R. & Attewell, P. (1993). Electronic Mail and Organizational Knowledge. Working paper, Department of<br />

Social and Decision Sciences, Carnegie Mellon University.<br />

Liang, D., Moreland, R. & Argote, L. (1995). Group versus individual training and group performance: The<br />

mediating role of transactive memory. Personality and Social Psychology Bulletin, 21, 384-393.<br />

Moreland, R. L. & Myaskovsky, L. (2000). Exploring the performance benefits of group training: Transactive<br />

memory or improved communication. Organization Behaviour and Human Decision Processes, 82, 117-133.<br />

Preece, J. (2000). Online Communities: Designing Usability and Supporting Sociability. John Wiley & Sons.<br />

Resnick, P. (2002). Beyond bowling together: SocioTechnical capital. In HCI in the New Millennium, edited by<br />

John Carroll. (647-672.) New York: Adddison-Wesley.<br />

Rogers, E. (1995). The Diffusion of Innovations. Free Press, New York.<br />

Rubin, K. & Goldberg, A. (1992). Object behaviour analysis. Communications of the ACM, 35 (9), 48-62.<br />

Scardamalia, M. (2000). Principles of Knowledge Building. Fourth Knowledge Building Institute, Toronto,<br />

Ontario, August 9-12, 2000.<br />

Scardamalia, M. & Bereiter, C. (1996). Rethinking learning. In Olson, D. R. & Torrance, N. (Eds.), The<br />

Handbook of Education and Human Development: New Models of Learning, Teaching and Schooling,<br />

Cambridge, MA: Basil Blackwell, 485-513.<br />

Sha, L. & Van Aalst, J. (2003). An application of social network analysis to knowledge building. Paper<br />

presented at the Annual Meeting of the American Educational Research Association, Chicago, April 21-25,<br />

2003.<br />

Suthers, D. (<strong>2006</strong>). Technology affordances for intersubjective meaning-making: A research agenda for CSCL.<br />

To appear in International Journal of Computer Supported Collaborative Learning, 1 (3).<br />

Valente, T. (1995). Network Models of Diffusion of Innovations. Hampton Press, Cresskill, NJ.<br />

Wasserman, S. & Faust, K. (1995). Social Network Analysis: Methods and Applications. Cambridge: Cambridge<br />

University Press.<br />

Wellman, B. (1996). For A Social Network Analysis of Computer Networks: A Sociological Perspective on<br />

Collaborative Work and Virtual Community. Center for Urban and Community Studies, University of Toronto,<br />

Canada.<br />

76


Weber, M. (2000). Modeling Technological Change under Uncertainty in Integrated Assessment Models for<br />

Climate Change, retrieved January 7, 2004 from http://www.mpimet.mpg.de/~weber.michael/modeling_tc.html.<br />

Wenger, E. (1998). Communities of Practice: Learning, Meaning and Identity. Cambridge: Cambridge<br />

University Press.<br />

Wenger, E., McDermott, R. & Snyder, R. (2002). Cultivating Communities of Practice: a Guide to Managing<br />

Knowledge. Boston: Harvard Business School Press..<br />

77


Donoghue, S. L. (<strong>2006</strong>). Institutional Potential for Online Learning: A Hong Kong Case Study. Educational Technology &<br />

Society, 9 (4), 78-94.<br />

Institutional Potential for Online Learning: A Hong Kong Case Study<br />

Sue L. Donoghue<br />

17A Island View, Midvale Village, 21 Middle Lane, Discovery Bay, Lantau Island, Hong Kong<br />

Tel/Fax: +852 28161607<br />

beldon@netvigator.com<br />

ABSTRACT<br />

Hong Kong’s tertiary education environment has changed dramatically in recent years with universities<br />

now facing specific educational challenges in the areas of critical thinking, ‘life-long learning’ and English<br />

language. The question exists as to what pedagogic developments will best allow the universities to address<br />

these challenges. In this paper, the appropriateness of online pedagogy as one response to these challenges<br />

is assessed through use of illustrative case-examples and post-course surveys. The potential of The<br />

University of Hong Kong (HKU) to implement this pedagogy is examined through a systematic<br />

consideration of the requirements for embedding online learning, specifically student and institutional<br />

knowledge, culture, and resources.<br />

The case examples provide empirical evidence to suggest that online environments are useful in addressing<br />

these challenges, largely as a consequence of afforded flexibility in teaching and learning, support of more<br />

student-centered learning approaches, and a high degree of student engagement. Opportunity for<br />

international collaborative teaching, with associated benefits in curriculum extension, cost-spreading and<br />

benchmarking, is also demonstrated. Within HKU, student factors appear to pose no major constraint to<br />

such development, but there exist significant inconsistencies in terms of institutional culture, pedagogic<br />

knowledge and non-hardware resources. The author concludes that small-scale online developments are<br />

viable and the cost of establishing and maintaining these need not be prohibitive. While online learning<br />

holds promise for HKU, the University will require internal institutional change to fully realize this<br />

potential.<br />

Keywords<br />

Online learning, E-learning, Pedagogy, Strategic planning, Higher education, The University of Hong Kong<br />

Introduction<br />

The tertiary education environment of Hong Kong (HK) has been subject to significant flux over the last 15<br />

years, as a consequence of evolving political and educational objectives. A number of challenges, including<br />

massification of undergraduate student numbers and changing graduate skill requirements, require universities to<br />

re-examine the nature of teaching and learning.<br />

Given these challenges, and the pervasive and integral use of personal information technology in HK society, it<br />

is timely to examine the potential of online learning as an appropriate response to the changing HK tertiary<br />

environment. Much has been made of the synergistic potential between online learning and the growing<br />

educational emphasis on ‘life-long learning’, with its intrinsic demands for flexibility (e.g. Ryan, Scott, Freeman,<br />

& Patel, 2000, p. 10).<br />

This paper examines, through empirical illustrative case study, the degree to which The University of Hong<br />

Kong (HKU), Hong Kong’s pre-eminent tertiary institution, is positioned to embrace online learning. There are<br />

two considerations to this: firstly, whether online learning environments, through their design, may offer<br />

opportunities to help address the specific challenges faced, and secondly, what is tangibly required by both the<br />

university and the student body to embed online learning environments.<br />

The Hong Kong tertiary environment: educational challenges<br />

Massification<br />

Immediately prior, and following the 1989 Secretary for Education and Manpower education strategy, there was<br />

a veritable ‘explosion’ of tertiary student numbers in HK (UGC, 1996). While many nations struggle to<br />

accommodate slowly increasing numbers of tertiary students as a result of societal evolution toward service<br />

economies and ‘qualification deflation’, such expansion was overshadowed in HK by active government policy<br />

to rapidly and extensively increase the percentage of students continuing into the tertiary sector. First-year firstdegree<br />

places (full time equivalents) at University Grants Committee (UGC) funded institutions increased from<br />

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the<br />

copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies<br />

are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by<br />

others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior<br />

specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.<br />

78


approx. 4000 to over 15000 in the decade from 1985/86 to 1995/96 (UGC, 1996). Further expansion is planned<br />

to bring the total number to around 55000 (HKSAR, 2000, par. 66).<br />

The HK Government is in the process of changing the educational structure from the current five-year junior<br />

secondary, two-year senior secondary, and three-year university (‘5+2+3’ structure) to a ‘3+3+4’ structure<br />

(HKSAR, 2004). This change in educational boundaries will replace the final year of secondary education with<br />

an additional year in undergraduate tertiary programs, resulting in a further expansion of numbers with a large<br />

influx of younger, less mature tertiary students.<br />

Coinciding with this massification, the HK Government is seeking to limit public purse burden by reducing the<br />

UGC recurrent grants to the tertiary sector. Between 1998 and 2004, the UGC realized a HKD 1.1 billion<br />

reduction in funding (UGC, 2004).<br />

Skill demands<br />

Students entering tertiary undergraduate programs are almost exclusively a product of the local secondary<br />

education system. This system is generally regarded as instilling a limiting teacher-directed, assessment-oriented<br />

culture within the student population. It focuses heavily on the lower, convergent cognitive levels of Bloom’s<br />

(1956) taxonomy of educational objectives (Bloom & Krathwohl, 1956), and fails to develop the higher,<br />

divergent levels (analysis, synthesis, evaluation); “Hong Kong education has always been regarded as<br />

examination-oriented and students were taught only to memorize and not to think.” (Fok, 2002, p.83).<br />

The UGC-commissioned Preparation of Students for Tertiary Education Report (POSTE) in 1996, found that<br />

“the aims of the Curriculum Development Council for inculcating creative thinking and rational and independent<br />

decision making are unlikely to be realized.” (UGC, 1996, p.2). The HK Government has determined that this<br />

represents a serious educational deficiency that needs to be addressed, and has accordingly attempted to<br />

introduce resources and remedial initiatives (HKSAR, 2000). The development of a ‘liberal education’<br />

promoting life-long learning and critical thinking presents a significant challenge to higher education<br />

institutions, and their traditional ‘divorced lecture’ pedagogy.<br />

English language proficiency also remains a concern for the HK Government, given its role as lingua franca in<br />

both academic and commercial environments, and the recognition that adequate numbers of multilingual<br />

graduates are of great importance to HK’s economy (UGC, 1996). The University of Hong Kong is officially an<br />

English-language institution, and instruction remains significantly in this language.<br />

The POSTE report (Cheng, Lai, Lam, Leung & Tsoi, 1996), concluded that “although English is the declared<br />

medium of instruction in many secondary schools, few of them earnestly teach in English” and that “A-level<br />

subjects are taught in Cantonese in the great majority of schools”. This language disjoint was amplified from the<br />

commencement of the 1998 academic year when the majority (307) of government and government-aided<br />

secondary schools in Hong Kong were officially converted to dominantly ‘mother-language’ teaching in<br />

Cantonese, with only 114 ‘high-achievement’ schools being permitted to pursue English-medium instruction<br />

(Lao & Krashen, 1999, p.1). The impact of this on general education levels remains a subject of debate, although<br />

it is likely the average level of exposure to English has been significantly further reduced at secondary level.<br />

Chen (2001, p.2), notes that ‘many students reach university unable to construct a single sentence correctly. It is<br />

disastrous’. It is clear, even without examining internal institutional language use that at least the ‘input’ side of<br />

the University language equation has changed dramatically. Cheng, Lai, Lam, Leung & Tsoi’s (1996) POSTE<br />

Report provides a summation of this change in student profile:<br />

The ‘cream’ which used to be admitted to local universities are [sic] now most likely attracted by overseas<br />

institutions. In other words,<br />

a) the students who are now in our universities are very different in nature from those who entered the<br />

universities 15 years ago;<br />

b) the universities are now facing a large number of eligible students whereas they used to face a small number<br />

of elite.<br />

The immediate cost of tertiary education on the individual student has greatly increased over the last decade.<br />

Composition fees for a fully government-funded bachelor’s degree currently exceed $42,000 p.a. and are<br />

scheduled to rise to $50,000 p.a. A result of this partial movement toward ‘user-pays’ education is a need for<br />

many students to have regular term-time employment simultaneously with their studies, or alternatively to<br />

79


contemplate part-time, rather than full-time, study. The need to work impacts tremendously on the students’<br />

ability to participate in the full gambit of campus-based learning activities. Already, teachers are witnessing<br />

lower attendance rates in lectures, receiving more requests from students to extend deadlines for submitting<br />

coursework, and observing reduced student participation in campus-based activities (Donoghue, 2002).<br />

Many students are now choosing to remain living in the family home, rather than the halls of residence. This<br />

reduces costs, as well as providing refuge from the extra pressures (principally the social commitments) that life<br />

in a university hall brings. Associated with this trend are longer travelling times (often 2-3 hours per day) and a<br />

reluctance to come to campus unless absolutely necessary. HK students are now more ‘mobile’ than before, and<br />

the teaching and learning environment must change to accommodate this trend (Donoghue, 2002)<br />

The HK tertiary environment can thus be characterized by:<br />

massively increased tertiary student numbers over the early 1990s<br />

the likelihood of further rapid expansion and introduction of younger students<br />

demand to reduce per unit costs<br />

students with an instilled rote-learning philosophy of education<br />

students with poor English-language skills, and of a potentially declining standard<br />

students requiring time and location flexibility to allow for other demands such as paid employment and<br />

residence location.<br />

Online learning: a potential response to these challenges?<br />

Online learning offers potential benefits in addressing the changing profile of the student body and the other<br />

significant challenges to the Hong Kong tertiary education sector, especially given what is often reported as the<br />

top three skill requirements of graduates – communicating in English, problem-solving, and information<br />

technology (see, for example, Fok, 2002).<br />

In terms of language, online learning offers second-language students opportunities for reviewing and re-reading<br />

material for full comprehension, placing all material into an overall context, and actively interacting<br />

asynchronously with teachers and peers, both local and foreign, in the second language.<br />

Much has been written of the flexibility in teaching and learning (removing time/place constraints) and more<br />

effective cognitive learning (and therefore ‘quality learning’) available through online learning. The integration<br />

of face-to-face and online learning to create ‘blended’ learning contexts has been seen to provide the<br />

independence and control required for students to develop critical thinking (Rosenberg, 2001; Stokes, 2000;<br />

Garrison & Kanuka, 2004). Once established, online learning, with its capability for large groups,<br />

internationalisation, and low capital item demands, has the potential for the high efficiency and low unit costs<br />

being currently sought in the tertiary sector.<br />

Scales of development<br />

In the last decade, many larger universities embarked on grand global online learning schemes that attempted to<br />

offer entirely online institutions (‘comprehensive virtual universities’), paralleling and competing with<br />

traditional residential university teaching (e.g. UK eUniversities Worldwide [UKeU], NYU Online, Scottish<br />

Knowledge, Fathom). These were often for-profit collaborations with commercial partners, with emphasis on<br />

capturing a ‘ground floor market entry’ into a perceived imminent paradigm shift in education delivery.<br />

‘First generation’ virtual universities were characterized by massive and often unexpected capital expenditure<br />

requirements, poor pedagogy, and an indefinite marketplace - “mistakes from which the industry is still<br />

struggling to recover; mistakes that had an undeniably negative impact on all key user groups in the e-learning<br />

marketplace” (Williams, 2003, p.1). The recently reported ‘salvage’ restructuring of the UKeU, originally<br />

backed by £55m of UK government funds, provides a typical example (see, for example, Naughton, 2004).<br />

Equivalently spectacular ‘failures’ were seen in the USA, with the US$40 million inter-institutional ‘Fathom’<br />

initiative headed by Columbia University, and the ₤9 million Open University’s foray into the American market<br />

(MacLeod, 2004). Such initiatives are likely to remain white elephant developments in the near future, and<br />

innovators in this area will suffer the significant costs of both product and market development.<br />

80


Such comprehensive parallel virtual-institutional development is of questionable relevance to a traditional<br />

residential campus university like HKU, whereas selected incorporation of online learning at course level within<br />

the existing university structure may offer rewards. The Vice-Chancellor of Kingston University summarises:<br />

Objectives<br />

It is not clear that universities are well equipped for it [establishing virtual university developments]…<br />

but universities are reluctant to be relegated to the supply chain. We want to be on top [as<br />

initiators/owners], not on tap [as providers]…There is a world of difference between developing oncampus,<br />

or near-campus, e-learning, and venturing into the global e-learning business. The first does<br />

not challenge the core purposes of the university; the second may. (Scott, 2001, p.1).<br />

There are two objectives to this exploratory examination. The first is to confirm through illustrative case study<br />

that online learning environments, at course level, have the potential to address some of the education challenges<br />

facing HKU. Of particular relevance are the development of skills required for life-long learning (especially with<br />

regard to communication and critical thinking), and the provision of flexibility in teaching and learning in terms<br />

of when and where teaching and learning takes place (in response to the changing student profile and numbers).<br />

The second objective is to provide some empirically-based commentary as to where deficiencies exist that<br />

impinge on HKU’s ability to embed online teaching and learning in response to these educational challenges.<br />

Methods<br />

The Case Examples<br />

Illustrative case example online courses were established by the author to provide teacher and student experience<br />

with online teaching and learning environments across a range of course contexts. These courses have now run<br />

over several academic years.<br />

Three courses in earth and environmental sciences have been developed, each with their own specific teaching<br />

and learning objectives and requirements for student and teacher engagement in the online environment.<br />

Importantly, in all courses, the use of technology is subservient to the pedagogical function. The online<br />

component is intended to promote ‘student-centred’ learning, involving both independent learning and the<br />

setting of learning goals by the student, and skills acquisition and development, specifically communication<br />

(teacher–student and student-student interaction), critical thinking, synthesis and language skills. All courses<br />

feature inbuilt synchronous (forum) and asynchronous (chat rooms) communication tools. Course assessment is<br />

tied to measurable outcomes in these areas and across all course components.<br />

Hong Kong’s Natural Environment (HKNE), is offered as a trans-faculty ‘broadening’ course, and was first<br />

made available in 2002-2003. Broadening courses were introduced as a compulsory curriculum element to<br />

provide students with non-specialist exposure to studies outside of their degree programme.<br />

HKNE utilizes ‘blended’ teaching and learning environments, involving online and field-based study, supported<br />

by classroom seminars. In this course, online study involves independent examination of web-provided course<br />

content and parallel participation in group-based discussions using the course forum (bulletin board). The<br />

discussion topics are set by the teachers and require that students use other resources to support their learning of<br />

these. The teacher’s role in the online discussions is facilitative. An intensive 3-day field camp provides<br />

experiential learning about the HKSAR environment and field skills development that is otherwise unavailable<br />

by either traditional lecture-based or exclusively online pedagogies. The enrolment quota for this course has<br />

been increased from and initial 45 students in 2002-2003 to 90 students in 2004-2005, reflecting teacher<br />

confidence in the effectiveness of the modes of instruction.<br />

Geohazards is a modular short-course developed through international teaching collaboration with Massey<br />

University (NZ). It was first made available in 2000-2001 and has run annually. Since its inception, it has<br />

undergone two revisions (of structure and content) based on teacher and student feedback.<br />

Geohazards is offered to three distinct groups of students in a dominantly ‘distance education’ mode, with<br />

physical resources for extension and support tutorials being made available through the teachers in their<br />

81


espective institutions. The primary groups are final year Earth Science undergraduate students at HKU (30<br />

students) and, separately, extramural (Diploma in Civil Defence) students at the collaborating institution<br />

(Massey University, New Zealand). Given its accessibility in terms of both content and pedagogy, the course has<br />

also been offered in past years to HKU medical students (MBBS degree; 25 to 30 students) as a Special Study<br />

Module for completion over their summer vacation. In this course, the assessment methods have been carefully<br />

chosen to allow examination of the learning that has taken place in the online environment. They include<br />

participation in group-based discussions using the course forum (bulletin board), synthesis essays, and research<br />

posters or companion website projects. For third year earth sciences students who regularly attend campus for<br />

other courses, viva voce-type interview assessment is also used.<br />

Volcanic Processes, the most recently developed course, is also a modular course developed through<br />

international teaching collaboration with Greenwich University (UK). This online course was first made<br />

available to HKU students (30 students) in 2003-2004. It utilizes both distance-education and classroom-based<br />

teaching and learning modes. The theory content and exercises are made available online. Students engage in<br />

independent learning of content and group-based discussions on set-topics using the course forum (bulletin<br />

board) with the teacher (who resides at Greenwich University) facilitating discussions from a distance. At a later<br />

date the teacher visits HKU to run a series of condensed workshops (assessed) with HKU students using the<br />

web-course as the basis for content.<br />

Access for all three courses is through a registered collective domain (www.e-HKU.net) hosted on an external<br />

commercial (US) provider at a sparing cost of around USD10/month. This allows easy, common access and<br />

equally rapid transmission for all three collaborating institutions.<br />

Course development is not static, but followed a purposefully reflective iterative approach (Figure 1), with<br />

subsequent course development and modification being driven by post-course reflection, significantly directed<br />

by data obtained from the students. For most students involved, the author’s courses have provided them with<br />

their first foray into online learning.<br />

Post-course surveys<br />

Figure 1. Reflective iterative development approach used<br />

Data from students was obtained by the use of post-course surveys. Survey questions were categorised into six<br />

specific areas of focus: student skills upon enrolment, technology availability, perceptions of different learning<br />

environments, value-added attributes of an online learning environment, pedagogy, and future course<br />

development.<br />

Given the mix of quantitative and decision/acceptance data being solicited, questions were largely presented<br />

through a Likert-type format (Trochim, 2002) which allows easy interrogation of both types of data. The surveys<br />

conclude with an open-ended comments request allowing free commentary on non-defined aspects.<br />

Survey forms are sent to students by email, with the choice of return by email attachment or printing and<br />

posting, should anonymity be desired.<br />

82


Determination of requirements<br />

The second objective requires a determination of the requirements for introducing online environments, and an<br />

interpretation of the HKU context with respect to these.<br />

The implementation of online learning is dependent upon three general determinants – knowledge, reluctance<br />

(culture), and resources (see, for example, McCormack & Jones, 1998). Knowledge includes the three aspects of<br />

subject content, pedagogy and technology. The cultural determinant refers to attitudes of participants, whilst<br />

resources embraces both tangible (e.g. hardware) and intangible (e.g. time) requirements. By presenting these as<br />

a matrix in which each parameter is examined in terms of both sides of the ‘education equation’ (the student<br />

body and the institution dimensions), and its consistency with the other parameters, a framework that allows<br />

systematic examination of HKU’s potential for online learning development (Table 1) can be produced.<br />

Knowledge<br />

Table 1. Determinants in the implementation of online learning environments<br />

Institution and Faculty Student Body<br />

Subject content<br />

Appropriate pedagogy<br />

Technical competency<br />

Technical competency<br />

Culture Teaching culture Learning culture<br />

Resources<br />

For course development<br />

and maintenance<br />

For access and use<br />

Relevant data from the student body dimension is obtained through the post-course surveys. Practitioner<br />

experience and literature (including publicly available quantitative data on resources) provides the basis for<br />

interpreting the institutional and faculty dimension.<br />

Observations<br />

Relevance of online learning to HKU educational challenges<br />

Flexibility in teaching and learning<br />

One distinct benefit of the online environment has been the ability to overcome time/place constraints on<br />

teaching and learning. This allows the courses to be available to wider, and in some cases larger, groups of<br />

students (across faculties), and involve teachers from other departments (as with HKNE) and institutions (as with<br />

Geohazards and Volcanic Processes).<br />

Characteristics of the online learning consistently rated as beneficial by the responding students across all<br />

courses are: being able to progress through the course content at their own speed (i.e. intensity flexibility), (98%<br />

MBBS, 82% HKNE, 80% Geohazards); being able to work from home (location flexibility), (100% MBBS, 84%<br />

HKNE, 73% Geohazards); and having the whole content available at all times (overview and context and selfpaced<br />

study), (100% MBBS, 90% HKNE, 80% Geohazards). “The web-based teaching method allows me to<br />

work in the most appropriate time and therefore I think it’s a good way to learn this course” [HKNE student,<br />

2003-04].<br />

It is good to have reading material on line since sometimes we don’t know what the lecturer is talking<br />

about and just fall asleep or daydreaming [sic]. The materials on the web allow us to read it at own<br />

speed and ask question through e-mail or ask the lecturer directly. Thus whether the lecturer is available<br />

for consulting or not is very important in online courses. [Geohazards student, 2000-01]<br />

The desire for this flexibility is demonstrated by the end-of-course surveys that indicate access is typically<br />

dominated by the 6pm-midnight and midnight-6am periods and from home or off-campus locations. Only onefifth<br />

of access is during normal working hours (6am-6pm), (see Figure 2). The ‘chat room’ function<br />

(synchronous communication), being inconsistent with time flexibility, is also less supported than the forum and<br />

email functions (asynchronous communication). Not surprisingly, the majority of HKNE students (82%) realize<br />

83


greater flexibility in course selection as a result of replacing rigidly time-tabled lectures with web-provided<br />

teaching.<br />

I think the site is a good tool for us to study. Because I think as a University student, we have to be<br />

more independent in learning! And the advantages of the site can be best seen recently. It is because it<br />

was least affected or totally unaffected by the suspension of classes. Thank you. [HKNE student, 2002-<br />

03]<br />

Figure 2. Illustrative access parameters for Hong Kong’s Natural Environment 2003-04<br />

[Data suggest the utility of a flexible learning environment]<br />

The advantage of the flexibility and accessibility afforded by online learning environments in the life-long<br />

learning context is obvious. This was clearly demonstrated not only by the ability for Geohazards to continue<br />

during the suspension of classes during the SARS outbreak in HK, but also the successful offering of this course<br />

to HKU medical students over the summer break outside of the teaching year.<br />

Language and Communication<br />

HKU students no longer receive the same exposure to English language as a result of the change in HK<br />

sovereignty and the consequent introduction of Chinese medium of instruction in secondary schooling in 1998.<br />

The promotion of Putonghua as an alternative ‘second language’ in HK, and the reduction in first-language<br />

English lecturing staff at HKU have also been influential in this regard.<br />

In addressing English language skills of students, there are two related considerations –increasing engagement of<br />

students in the use of English language and improving comprehension of subject content in English.<br />

The traditional didactic teaching context provides little assistance in either aspect; it gives very limited<br />

opportunity for active student engagement in the language, and the verbal lecturing presentation of specialist<br />

content is often difficult for a second-language listener. The author’s post-course surveys indicate that a<br />

significant percentage of students (40%) describe their comprehension of English in traditional verbal lectures as<br />

“less than 80% understood”. From this case study, online courses demonstrated potential benefits in both<br />

engagement and comprehension aspects.<br />

The author’s experience is that for some students, communicating verbally in English is an uncomfortable<br />

experience. They find expressing thoughts and phrasing questions verbally in English intimidating. Developing a<br />

‘learning partnership’, even in the small-group classroom setting, with this type of student is difficult, as they<br />

84


tend to remain reticent and reluctant to engage, relying on friends to phrase questions and seek responses on their<br />

behalf. In the online environment, however, asynchronous communication tools allowed these students to engage<br />

confidently. They could ‘prepare’ their questions and responses before posting these, helping them to build<br />

confidence in communicating with the teacher and classmates.<br />

Informal involvement (i.e. not prescribed) in online discussion was universal, and many students took advantage<br />

of the real-time chat room to relate with the teacher. The benefits of this were immediate, with students who had<br />

interacted through the course chat rooms the night before actively seeking out the teacher on campus for<br />

continued discussion, in a semi-mentoring role.<br />

In terms of comprehension, many students in their post-course surveys have remarked on the several benefits<br />

that a web-based format allows over a traditional lecture presentation: an unforced speed of presentation (reading<br />

rather than listening), an ability to repeat and review content when comprehension is not immediate, and an<br />

ability to ‘overview’ and ‘context’ material through having the complete content available at all times.<br />

The post-course surveys for Geohazards and Volcanic Processes (courses which involve international<br />

collaboration, and in the case of Geohazards, course operation in full distance education mode) indicated a<br />

surprising student willingness (ranging from 56-73%) to have work assessed by the overseas (English-language)<br />

teacher. The potential also exists for asynchronous group interaction with first-language English students and<br />

teachers operating in parallel at these institutions. Not only would these features support engagement (and<br />

therefore learning) and language development, but they would also provide a subsidiary benefit in providing<br />

external, international ‘quality-benchmarking’ of curricula and student performance.<br />

I believe the web-based teaching gave me control over my learning, which was most important. I thought<br />

there could be more interaction with overseas students/teachers, e.g. perhaps include some of them into<br />

our small discussion groups. This is the first time I’ve done a course like this, and it’s been a very positive<br />

experience for me. Thanks. [MBBS Geohazards student, 2002-2003]<br />

Critical-thinking and problem-solving<br />

Critical-thinking and problem-solving skills can also be developed in online learning environments, although the<br />

extent to which this is supported depends largely on the design of the course, i.e. the specified learning<br />

objectives and the assessment methods used to determine learning. This is also the case with traditional<br />

classroom-based teaching.<br />

All case example courses were designed to promote ‘enquiry-based’ learning. All required student participation<br />

(assessed) in online peer-group discussions, necessitating self-directed study of online content and other learning<br />

resources (e.g. DVD, periodicals) and the reporting of ‘synthesis’ results (that evidence acquisition and<br />

application of new knowledge) via the course forum. This learning was supported by high degree of access to a<br />

‘virtual community’ of learners and teachers, providing for targeted and timely feedback. “I think the course is a<br />

well-organized one and I particularly like the way we can contact and obtained [sic] help from the teachers,<br />

either using a forum or thru [sic] email” [MBBS Geohazards student, 2002-2003]. This accessibility provides for<br />

a more student-centred approach to learning, which is integral to life-long learning, and which is not supported<br />

by the traditional ‘divorced lecture’ pedagogy.<br />

Critical-thinking and problem-solving skills are further developed in the ‘blended-learning’ environments of<br />

HKNE and Volcanic Processes which have a high degree of student engagement in the learning process (through<br />

field-based and classroom activities). In the HKNE course surveys, responding students rated field trips and the<br />

online environment (84% and 76%, respectively) consistently above the traditional environments of lectures<br />

(44%) and practical classes (36%) as the ‘teaching methods’ they felt were ‘helpful’ or ‘very helpful’ to their<br />

learning.<br />

Individual viva voce-type assessment was offered to third-year Geohazards students. Discussion with each<br />

student sought to determine if the course learning objectives had been met through largely independent learning<br />

and skills (e.g. critical-thinking) further developed. This type of assessment proved surprisingly popular,<br />

especially given that, across all courses and all student groups, consistently 33% or less of students regard<br />

themselves as confident users of oral English language. “Oral test is quite new to me. But I think it is a very<br />

good experience. Before I took the oral test, I was very nervous. After I took it, I feel more comfortable and<br />

think it is a good testing method!” [Geohazards student, 2003-04].<br />

85


Cost-benefit<br />

One distinct benefit of online courses is their scalability in terms of access, and therefore the ability to ‘teach’<br />

increasingly larger class sizes with little additional development cost. Although scalability is an important<br />

consideration in assessing cost-effectiveness of online course developments, the fixed costs for course<br />

development have to be met regardless of class size.<br />

The case example courses were developed within the author’s earth sciences specialism with its small student<br />

numbers, ranging from 30 students in Geohazards to 90 for the interfaculty HKNE broadening course. By most<br />

standards, these developments are small-scale and therefore are likely to present the least-favourable cost-benefit<br />

circumstance. That said, all three courses were established ground up for a modest total expenditure of around<br />

USD50,000 demonstrating that small-scale developments are viable and that the cost of establishing and<br />

maintaining online environments need not be prohibitive.<br />

The author’s department, the Department of Earth Sciences (DES), is the only tertiary department offering an<br />

Earth Sciences degree programme in HK. With an annual freshman enrolment of less than 30 students, it is too<br />

small to feasibly offer the desired range of specialist courses for an integrated and comprehensive curriculum<br />

internally. In the case of both Geohazards and Volcanic Processes, international collaboration importantly<br />

allowed for development of desired expert content at a slight fraction of the otherwise untenable cost of regular<br />

internal employment of such specialists.<br />

Generalisability<br />

While both enrolment numbers and the centralized University end-of course ‘student evaluation of teaching’<br />

(SET) scores demonstrated the popularity and acceptance of the example case courses, the author’s post-course<br />

surveys indicate that this does not extend to a carte blanche affirmation of online learning as an alternative to<br />

traditional lecturing.<br />

There is, however, a consistency in students’ determination of those general categories of courses to which an<br />

online learning component would be suited. Typically greater than 80% of students identify ‘broadening<br />

courses’, ‘summer study modules’ and modular courses (like Geohazards) as suitable for online learning<br />

components. It could be inferred that such support reflects the students’ perceptions of the inherent benefits in<br />

terms of time and place constraints. ‘Language courses’, with their strong interpersonal and verbal components<br />

are universally identified as inappropriate for online methods. ‘Interfaculty electives’ are more mixed, with<br />

generally around 50% identifying these as appropriate for online learning development.<br />

Full semester core courses as a category typically find favour with less than 30% of respondents. There is,<br />

however, a great diversity in the nature of full semester core courses, and therefore perhaps a consequent<br />

unwillingness to assign all such courses as collectively suitable or unsuitable. Interrogation by specific course<br />

title yields a consistent picture. When asked to categorize a listing of other core Earth Science courses as to<br />

appropriateness for online learning, third year Earth Science Geohazards students have, over the last three years,<br />

consistently and dominantly (greater than 80% of respondents) graded the same three core courses as appropriate<br />

for online learning development, whilst deeming the remaining almost universally as inappropriate. It may well<br />

be a case of ‘customer knows best’, and it may be that student commentary on pedagogical design is undervalued<br />

and under-utilized.<br />

Feasibility of online developments given HKU’s current institutional status<br />

The extent to which the institution currently meets the requirements for the successful development and<br />

implementation of online learning is examined in terms of the determinants identified in Figure 2, which are<br />

knowledge (technical competency and pedagogy), reluctance or ‘culture’, i.e. the teaching and learning culture<br />

within the institution, and resources (provision and access).<br />

Knowledge<br />

It is apparent that in HK there is a high level of computer literacy, and student knowledge does not hinder the use<br />

of online learning. In the author’s post-course surveys, students reiterate that training is not required for them to<br />

86


effectively engage in online learning. While there are minorities who have a ‘fascination with technology’ and<br />

regard themselves as ‘advanced users’, the majority of students, typically around three-quarters, regard computer<br />

technology only as a tool, and themselves as but ‘competent’ users of computers.<br />

At the institutional level, knowledge of both technical and educational principles is required to construct an<br />

online learning environment. This combination of skills is one that few teachers have or might be expected to<br />

have. Presently, the three spheres of knowledge needed to create online learning environments (content,<br />

technology and pedagogy) remain noticeably divorced from one another within the institution.<br />

Culture<br />

Hong Kong society has been an ‘early adopter’ of internet technology, and this acceptance and use has been<br />

especially rapid and pervasive in the ‘educated’ sector of society. As early as 2002, a HK Government survey<br />

indicated 93.4% of all students above the age of ten were ‘internet users’ (HKCSD, 2003). In Hong Kong, there<br />

is clear acceptance, and even anticipation, of the growing role of internet technology in society (see ITU, 2003).<br />

In terms of acceptance of online learning specifically, it is worth noting that while it is consistent with the<br />

desired ‘educational culture shift’ toward student-centered, flexible, life-long learning based on critical thinking<br />

and personal responsibility, it does represent a significant departure from the surface-learning heritage of passive<br />

spoon-fed ‘threshold’ recall in the teacher-centered, examination-oriented secondary system (e.g. Yeung, 2005,<br />

p. E3). Practices that focus on continuing assessment, development of non-recall skills, “discourse of<br />

participation” (Li, 1999) and ‘out-of-the-box’ extension of concepts into non-familiar contexts can elicit some<br />

reluctance by a minority who regard formal tertiary education as a series of heavily guided and detailed shortterm<br />

assessment and content-based thresholds.<br />

The move to more student-centred learning approaches requires commensurate change in both the teachers’ and<br />

students’ attitudes to learning. For some students, this adjustment is difficult because of the greater responsibility<br />

placed on them for the learning that takes place. In all courses, there was an identified minority of students who<br />

attempted to replicate the familiar teacher-centered approach. This was illustrated by, for example, their<br />

attendance at course tutorials not to engage in discussion with teachers and peers, but in the hope they would<br />

receive a ‘lecture’, and their requests for highly specific exemplar assessment material. Such students found the<br />

flexibility of online learning a challenge to their personal time-management. “It is good to have the online<br />

material available all the time but when there are lots of other assignment (e.g. tutorial work in other courses),<br />

working online will be put at a later priority” [HKNE student, 2003-04].<br />

For these students, these observations suggest a distinct preference for definite threshold instruction and a<br />

dictated time-management environment when the ‘stakes are higher’ and a consequent reluctance to assume a<br />

high-degree of personal responsibility for the learning that takes place. The motivating desire appears to be ‘not<br />

to fail’ rather than ‘to learn’. For the majority, the ‘leap’ to more student-centred environments is welcome, with<br />

feedback indicating they perceive this environment as beneficial to their personal and academic development.<br />

At the institutional level, such developments require that teachers are knowledgeable in basic educational<br />

principles, specifically how students best learn. They must be willing and able to invest time and effort in teacher<br />

professional development and consequent development of practice. Teachers need to be supported by an<br />

institutional culture that recognizes and rewards teaching as a primary role of the university.<br />

The author’s experience, in her recent role as Faculty Teaching & Learning Network (TLN) Consultant, suggests<br />

the dominant institutional culture at HKU regards ‘education’ as a subordinate university function. By way of<br />

illustration, a TLN survey of the Faculty’s teachers identified that 38% believed teaching performance has no<br />

impact on promotion prospects, and almost all (94%) of the respondents indicated that their teaching has not<br />

been recognized in any way at Faculty or University level. Almost all those surveyed indicated they spent more<br />

time on research than teaching roles. Not surprisingly then, less than 15% of Science teachers had engaged in<br />

teacher professional development activities or programmes offered by the staff development unit during their<br />

tenure at HKU. Less than 5% had prepared a teaching portfolio to document their development as a teacher.<br />

Teacher knowledge of documentation intended to guide teaching development (i.e. Departmental Development<br />

Plans [DDP’s] and the 2002 externally-reviewed Faculty Teaching and Learning Quality Process Review<br />

[TLQPR] submissions) was negligible.<br />

87


Resources<br />

The development of online learning at HKU does not appear limited by technological knowledge or resources,<br />

with required hardware and internet access being pervasively available, both on campus and within the<br />

community as a whole.<br />

In the University, the HKU/IBM Assured Access Mobile Computing Programme, which commenced in 1998,<br />

seeks to provide subsidized purchase of IBM notebook computers and software to incoming students. It has<br />

historically attracted around 80% of first-year full-time (FYFT) students (see Table 2), despite the fact that<br />

almost all such students have existing computer resources already available (HKUCAUT, 2004).<br />

Table 2. Computer hardware availability for first-year students at HKU. Data from HKUCAUT (2004)<br />

Percentage of HKU FYFT students taking up HKU/IBM Student Notebook Computer Programme<br />

1998/99 1999/00 2000/01 2001/02<br />

84.8 78.9 79.3 75.3<br />

Percentage of HKU first year students with computer ownership (by student or student’s family) before<br />

enrolment<br />

1998/99 1999/00 2000/01 2001/02<br />

92.0 96.4 99.0 99.3<br />

In addition to this private-ownership program, the University, with a full time enrolment of around 12,000,<br />

further records 5,630 ‘public’ computers (HKUERO, 2003).<br />

In concert with this, The University of Hong Kong has invested extensively in network infrastructure, both on<br />

campus and in the halls of residence, in order to achieve ‘ubiquitous’ network access. Hallnet ensures “all<br />

student residential halls are connected to the HKU Campus Network with a network point provided for each hall<br />

resident”, and the Access Everywhere Network (ACEnet), “provides extensive roaming network access for use<br />

by staff and students” on the remainder of the campus (HKUCC, 2004). The External Relations Office<br />

(HKUERO, 2003) records impressive figures of 22,220 network points, 10,320 PC connection points, 450<br />

wireless network access points, and 308 Mbps total bandwidth for direct connection to the internet available on<br />

the campus.<br />

Access to courses must also be readily available off campus, and in this regard, HK is exceptionally wellresourced.<br />

In the wider off-campus community, the Office of the Telecommunications Authority (OFTA)<br />

estimates there are in excess of 2.33 million registered internet accounts in Hong Kong (March 2004), excluding<br />

users ‘who are not customers of the licensed ISPs, such as users of the campus networks in the universities’.<br />

(OFTA, 2004a, p.3). This represents approximately one registered account for every three citizens.<br />

Further to the availability of internet connection, the typical bandwidth in HK is very high. As a result of the<br />

concentrated apartment block housing, over 95% of HK households have broadband (services with downloading<br />

speed of 1 Mbps or above) potentially available. This is complemented by the highest international internet<br />

bandwidth per capita in the Asia-Pacific region and near the lowest broadband connection fees, allowing pricing<br />

to become among the cheapest in the region (ITU, 2003). There has been a consequent rapid expansion and<br />

conversion to broadband in the HK market, with an 18-fold increase in broadband accounts in the 2000-2003<br />

period (OFTA, 2004b). By 2003, 83.7% of household internet connections were broadband (HKCSD, 2003).<br />

Such transition in bandwidth is mirrored by the ‘home’ connection shown in representative post-course surveys<br />

over the last few years, with the 2003 MBBS cohort the first to report 100% broadband access.<br />

In terms of comparative statistics with respect to internet resources, be it in terms of availability, bandwidth, or<br />

pricing, Hong Kong holds a pre-eminent place internationally, and is accordingly developing into a ‘broadband<br />

society’ where a pervasive range of societal functions (e.g. banking, travel, utilities, government) is becoming<br />

available online.<br />

With respect to the institution, HKU is extremely well-provisioned with the hardware and software infrastructure<br />

components for the ‘consumption’ of online learning. It does, however, suffer in the provision of time, training<br />

and support necessary to ‘implement’ such an initiative. A succession of short-lived central support centers, with<br />

their often ill-defined financing and independent service role, has historically been poorly utilized by individual<br />

88


teachers. Despite the significant cost of these centres on institutional resources, the development of individual<br />

courses through these centres is not centrally supported, but requires independent funding by the teacher.<br />

As early as 1999, the Secretary-General of the UGC questioned institutional commitment to the provision of<br />

these less tangible resources toward online learning development: “are institutions prepared to invest in the<br />

facilities and the training and staff development that will be required to enable faculty to make full use of the<br />

benefits of the technology without being overwhelmed by it?” (French, 1999, p.2)<br />

Discussion<br />

The empirical evidence provided by the successful and continuing operation of the case examples indicates that<br />

online learning does in fact have a role in addressing some of the education challenges facing the HKU<br />

environment.<br />

The degree to which online teaching and learning is an appropriate pedagogy is course and content specific. For<br />

some courses, it may be most beneficial when developed as one of a range of integrated teaching and learning<br />

approaches (i.e. ‘blended-learning’ environments) as in HKNE and Volcanic Processes. In other courses, it may<br />

be beneficial to utilize full ‘distance-education’ as in Geohazards. For some courses it may offer no specific<br />

benefits and be an inappropriate pedagogy. Students with previous exposure to online learning environments can<br />

usefully indicate where online learning may yield specific benefits.<br />

Developing the potential<br />

For HKU, online learning presents a potential consistency with the formally espoused strategic direction, with<br />

the University’s mission and planning priority statements emphasizing the development of a comprehensive<br />

education, life-long learning and internationalisation, and the need to deploy new technologies in advancing<br />

teaching, learning and scholarship (HKUSPU, 2003).<br />

For effective development and utilization of online learning environments, there must be a consistency between<br />

the determinants of knowledge, culture, and resources (Figure 3). Many historical developments have failed to<br />

deliver the benefits expected, largely as a result of major inconsistencies or ‘disconnects’ (Figure 3) between<br />

these three parameters.<br />

CULTURE<br />

KNOWLEDGE<br />

RESOURCES<br />

Figure 3. Consistency between parameters favourable for online development is represented by the shaded<br />

region; other areas indicate inconsistency in one or more of the parameters<br />

‘Knowledge’ inconsistencies arise when there is no clear mapping between pedagogical and technical<br />

knowledge, or where either is insufficient to allow effective development within the constraints of the other<br />

parameters. An example of this is when an institution’s online development is a response to the demand to be<br />

‘up-with-the play’ rather than being driven by pedagogy. This can result in the application of online teaching to<br />

89


courses where it is has little potential benefit, or use of the technology at an incorrect or superficial level. Such<br />

initiatives are often characterized by replication, where the technology maintains an existing teacher-centric<br />

pedagogy without change, or simple redundancy, where the online development is both supplementary and<br />

superfluous to the existing pedagogy.<br />

‘Cultural’ inconsistencies arise when the institution’s policies and practice do not support pedagogical<br />

developments, or where either teachers or students are unaccepting of it, despite the availability of both<br />

resources and knowledge. Many technically-robust and comprehensive ‘platform’ developments at universities<br />

have failed to solicit the appropriate utilization by those teachers and courses that could potentially benefit from<br />

these and do not establish ‘critical mass’ that might embed the pedagogy at an institutional level.<br />

‘Resource’ inconsistencies arise either when an institution lacks the physical resources to realize the<br />

developments possible given the available knowledge and culture, or where it is seduced by the “rapture of<br />

technology” (Ehrmann, 2002), and sponsors initiatives that are too resource-demanding in their development,<br />

evolution, and maintenance to have any hope of long-term continuation. These latter ‘bells and whistles’<br />

initiatives are characterized by focus on advanced technology rather than education, and consequently often<br />

evolve extended and convoluted structure, and large technology costs. Such initiatives require a high level of<br />

continuing technical support beyond that immediately available by or to the teacher. While being eminent<br />

‘institutional show-pieces’, they demonstrate excessive cost for limited benefit, and are typically either transient,<br />

with a lifespan tied to funding availability, or static after their initial development.<br />

At HKU, there appear no inconsistencies within the student body that present a major impediment to online<br />

development. It is apparent that the required technical literacy and technological acceptance of internet-based<br />

teaching is established. Students are quick to identify the advantages of this methodology and determine which<br />

courses would lend themselves to it. The reluctance of some students to assume the challenge of student-centered<br />

pedagogies may be expected to reduce with growing familiarity. Resource availability is exceptionally high.<br />

The major hindrances to the development of online learning at HKU appear to lie largely in institutional<br />

inconsistencies, particularly those of culture, pedagogic knowledge, and non-hardware resources.<br />

Institutional culture<br />

The ‘culture’ within an institution is significantly framed by its strategic and internal policies, and consequent<br />

reward structures. HKU is currently pursuing market ‘niching’ through elitism and ‘excellence’ largely<br />

dominated by quantified research publication, as have many ‘first-founded’ state universities worldwide. This<br />

developing and unambiguous research focus has strengthened a perception that teaching and education aspects of<br />

scholarship are not the primary focus in university career paths, dominantly rewarding discipline-based<br />

publication.<br />

Stoll and Fink (1996), provide a categorization of institutional educational cultures, based on a matrix using<br />

continua of both outcomes achieved and the changes in educational process applied (see Figure 4). It could be<br />

asserted that HKU traditionally is placed in the scope of a cruising institution, i.e. one that possesses a<br />

historically high ability intake where the students have achieved largely in spite of teaching quality. Rankings<br />

based on absolute achievement in alternative criteria, such as research publication lists or inter-institutional<br />

advantage in securing state research monies, rather than ‘value added’ education, often give the illusory<br />

appearance of overall institutional effectiveness.<br />

Figure 4. Educational culture: modified from Stoll & Fink (1996, figure 6.1, p.85)<br />

90


Online developments require a large initial investment of time and application by the teacher, and presently, a<br />

consequent perceived cost in terms of career development.<br />

I guess it will take a long time to convince the average junior lecturer or assistant professor that their<br />

track record in research is not much more important for their career advancement than developing a truly<br />

imaginative and exciting web-based learning module for one of their department’s courses (French, 1999,<br />

p.7)<br />

Without a significant Kuhnian ‘paradigm-shift’ in institutional policy and culture to offer balance and promote<br />

the concept of HKU as a value-adding educational institution, the future of teaching-based initiatives, including<br />

online learning, will continue to be one of fragmentary, non-integrated and transient efforts by “pedagogical lone<br />

rangers” (D. Johnson, pers. comm.) achieving “random acts of progress” (L. Horne, pers. comm.).<br />

Pedagogic knowledge<br />

It is imperative that teaching developments be founded in an appreciation of pedagogical advantage, rather than a<br />

fascination with technology or academic politics. Lee (2004) points out the required direction of development,<br />

stating “… the only way to get maximum benefit out of educational technology is to start with the pedagogy.”<br />

The development of the internet has given academia an opportunity to reassess the pedagogy of teaching<br />

in higher education. It has allowed us to return to some of the true principles of teaching, such as<br />

interaction, reflection, good resources, and student-centered teaching. It is interesting that students on<br />

online courses report that they are getting more human interaction than on any other type of course. (R.<br />

Mason, professor of educational technology at The Open University, United Kingdom quoted in<br />

Williams, 2003)<br />

‘Pedagogic knowledge’ is not generally a formal pre-requisite for university teaching. It has been said that<br />

university teaching is indeed the most eminent professional role that is placed in the hands of those formally<br />

unqualified in that function (T. Morrison, pers. comm.). This lack of qualification in a primary role of the tertiary<br />

education sector can be compounded when institutional policy and culture fails to effectively provide any<br />

incentive for engagement in reflective practice, and active improvement or professional development in that<br />

function.<br />

At HKU, human resource policy has largely failed to exact and reward commitment to professional teaching<br />

practice. The recent ‘re-inventing’ of the institution as ‘research-led’ (Yeung, 2004, p. E3), with an unambiguous<br />

focus on research publication, has aggravated the divide between teaching and research roles rather than<br />

sponsoring effective integration.<br />

Non-hardware resources<br />

Currently, teaching development within HKU is being driven by small and disconnected groups of individuals<br />

with funding through a ‘research-model’ of small discrete grants of limited tenure. There is little opportunity or<br />

institutional endorsement for successful teaching developments to be funded or developed as ‘going concerns’,<br />

with a regard for extension and integration into a university-wide context. Consequently, teaching initiatives,<br />

including those involving online teaching, tend to be “a series of attempts at creating new and alternative wheels,<br />

with little intent at placing the wheels on the cart to get it moving” (I. Bell, pers. comm.). Such a fragmentary<br />

approach also results in a diversity of initiatives being repeated, and a consequent “sandstorm of confused<br />

practice and promotion” (Moore, 1998, p.6).<br />

Both institutional policy and resource provision must recognize the ongoing nature of teaching, and support both<br />

the initial development and the subsequent continuation and potential extension.<br />

Resources, including funding and personnel (both technical and, more importantly, educational specialism),<br />

should be directed to embedding proven and sustainable pedagogies. The requirement for consistency, expertise,<br />

and avoidance of expensive replication, suggests that this may be best achieved through an accountable<br />

centralized unit, with focus on those courses identified as appropriate.<br />

91


A fellowship programme, such as the University of New South Wales ITET initiative described by Lee (2004),<br />

may offer peer-driven extension through the academic community and resulting embedding of pedagogic<br />

initiatives within the institution. Importantly, the inclusion of such programmes within policy signals an<br />

institutional commitment to recognising teaching and promoting a much needed shift in culture.<br />

Summary<br />

The last decade has seen HKU move into a context that demands the development of new teaching paradigms in<br />

addition to the traditional ‘sage-on-a-stage’ one-way teacher-centric lecturing. In many aspects, the limitations of<br />

traditional teaching are becoming significantly more pronounced with the rapid environmental changes in the<br />

HK tertiary sector that have placed this institution in an unfamiliar educational environment. Online education<br />

appears to hold tremendous promise for HKU in addressing many of the exceptional environmental factors that<br />

demand a greater education focus. The availability and acceptance of the technology in HK make it a feasible<br />

and viable development for the tertiary sector. Any policy toward online learning development should be<br />

compatible with these demands and also with the institution’s espoused mission and goals, strengths and areas of<br />

distinction (Lujan, 2002), and assessment of cost-benefits. In the case of HKU, it is clear that the institution<br />

wishes to continue as a pre-eminent bricks and mortar institution within dominantly state-funding. The<br />

‘university-replacement’ virtual institution, with its massive initial capital expenditure, is currently an<br />

inappropriate course of action. Globally the pursuit of such policies is likely to remain both expensive and<br />

unsuccessful in the immediate, and such developments are best left to institutions with an existing ‘open<br />

university’ or ‘extramural university’ experience base. Importantly, for HKU, with its history and credibility in<br />

society, online learning assumes the mantle of an evolutionary pedagogy with an established client base, rather<br />

than a contested ‘paradigm shift’ seeking to establish a market.<br />

Focused, niche initiatives, especially those leveraged by collaborative development amongst faculties and<br />

institutions at undergraduate level, offer clear and unambiguous benefits in addressing the changing demands of<br />

the peculiar HKU environment. In these situations, advantage may arise through either replacement or ‘blending’<br />

with traditional teaching pedagogies.<br />

Presently, however, HKU is not positioned to develop the online learning potential fully. While the level of<br />

readiness by the student body appears high, the institution has notable inconsistencies between knowledge,<br />

culture and resources. If HKU is to develop this latent potential in a manner meriting its espoused goals in<br />

‘world-class education’ and ‘life-long learning’, its policies must tangibly aim at shifting institutional culture and<br />

overcoming the inconsistencies identified.<br />

Acknowledgements<br />

The author acknowledges the teaching collaboration with Massey University, New Zealand and Greenwich<br />

University, and the funding support provided by The University of Hong Kong Senate Teaching Quality<br />

Committee and the University Grants Council (Teaching Development Grant). Thanks are extended to the<br />

reviewers for their thoughtful reviews of the manuscript.<br />

References<br />

Bloom, B. S., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: The classification of educational<br />

goals, Handbook I: Cognitive domain, London: Longman.<br />

Chen, E. (2001). Towards quality tertiary education in Hong Kong: A summary of the remarks by guest speaker<br />

Professor Edward Chen, President of Lingnan University. HKDF Newsletter, 18, retrieved March 14, <strong>2006</strong>,<br />

from, http://www.hkdf.org/newsletters/0107/0107_8.htm.<br />

Cheng, K. M., Lai, A. Y. Y. W., Lam, C. C., Leung, K. S., & Tsoi, H. S. (1996). POSTE: Preparation of<br />

students for tertiary education: Final report: A study commissioned by the University Grants Committee (4 th<br />

Ed.), Hong Kong: Hong Kong Government Printer.<br />

92


Donoghue, S. L. (2002). Online Teaching - An international approach. Proceedings of the Association of<br />

Southeast Asian Institutions of Higher Learning (ASAIHL) Seminar on Lifelong Learning, Nanyang<br />

Technological University, Singapore, 221-228.<br />

Ehrmann, S. C. (2002). Improving the outcomes of education: Learning from past mistakes. Educause,<br />

January/February, 54-55.<br />

Fok, S. C. (2002). Teaching critical thinking in a Hong Kong secondary school. Asia Pacific Education Review<br />

2002, 3 (1), 83-91.<br />

French, N. J. (1999). E-Education: Are Hong Kong’s higher education institutions rising to the challenge?<br />

Keynote address paper presented at the 5 th Hong Kong Web Symposium, <strong>October</strong>, Hong Kong.<br />

Garrison, D. R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative potential in higher<br />

education. The Internet and Higher Education, 7, 95-105.<br />

HKCSD (The Hong Kong Census and Statistics Department) (2003). Hong Kong as an information society<br />

(2003 Ed.), Hong Kong, China: Hong Kong Government Printer.<br />

HKSAR (The Government of the Hong Kong Special Administrative Region) (2000). Chief Executive’s policy<br />

address 2000, retrieved March 14, <strong>2006</strong>, from, http://www.policyaddress.gov.hk/pa00/eindex.htm.<br />

HKSAR (The Government of the Hong Kong Special Administrative Region) (2004). Chief Executive’s policy<br />

address 2004, retrieved March 14, <strong>2006</strong>, from, http://www.policyaddress.gov.hk/pa04/eng/index.htm.<br />

HKUCAUT (The University of Hong Kong Centre for the Advancement of University Teaching) (2004). 20<br />

Questions about the HKU/IBM student notebook computer programme, retrieved March 14, <strong>2006</strong>, from,<br />

http://147.8.151.130/itt/homepage/itt/2_HKU_IBM/20Q&A.<br />

HKUCC (The University of Hong Kong Computer Centre) (2004). HKU campus network, retrieved March 14,<br />

<strong>2006</strong>, from, http://www.hku.hk/cc/home/networks/hkunet.htm.<br />

HKUERO (The University of Hong Kong External Relations Office) (2003). The University of Hong Kong:<br />

Quik stats, retrieved March 14, <strong>2006</strong>, from, http://www.hku.hk/eroesite/html/pub/index_pub.htm.<br />

HKUSPU (The University of Hong Kong Strategic Planning Unit) (2003). The University of Hong Kong:<br />

Strategic development 2003-2008, retrieved March 14, <strong>2006</strong>, from, http://www.hku.hk/strategicbooklet/pdf/strategic-development_e.pdf.<br />

ITU (International Telecommunications Union) (2003). Broadband as a commodity: Hong Kong, China internet<br />

case study, retrieved March 14, <strong>2006</strong>, from, http://www.itu.int/ITU-D/ict/cs/hongkong/material/CS_HKG.pdf.<br />

Lao, C. Y., & Krashen, S. (1999). Implementation of mother-tongue teaching in Hong Kong secondary schools:<br />

Some recent reports. Discover, retrieved March 14, <strong>2006</strong>, from,<br />

http://www.ncela.gwu.edu/pubs/discover/05hongkong.htm.<br />

Lee, A. (2004). Institutional approaches to e-learning: UNSW and the ITET fellowship. HERDSA News, 26 (1),<br />

16-17.<br />

Li, M. S. (1999). Discourse and culture of learning - communication challenges. Paper presented at the joint<br />

AARE-NZARE Conference, Melbourne, Australia, retrieved March 14, <strong>2006</strong>, from,<br />

http://www.aare.edu.au/99pap/lim99015.htm.<br />

Lujan, H. D. (2002). Commonsense ideas from an online survivor. Educause, March/April, 29-32.<br />

MacLeod, D. (2004, April 13). The online revolution, mark II. Education Guardian, retrieved March 14, <strong>2006</strong>,<br />

from, http://education.guardian.co.uk/elearning/story/0,10577,1190470,00.html.<br />

McCormack, C., & Jones, D. (1998). Building a web-based education system, New York, USA: John Wiley &<br />

Sons.<br />

93


Moore, M.G. (1998). Introduction. In C. Gibson (Ed.), Distance learners in higher education: Institutional<br />

responses for quality outcomes, Madison, WI, USA: Attwood Publishing, 1-8.<br />

Naughton, J. (2004, March 21). A little e-learning is a dangerous thing. The Observer, retrieved March 14, <strong>2006</strong>,<br />

from, http://education.guardian.co.uk/elearning/comment/0,10577,1174384,00.html.<br />

OFTA (Office of the Telecommunications Authority, Hong Kong Government) (2004a). Statistics of customers<br />

of licensed internet service providers in Hong Kong, retrieved March 14, <strong>2006</strong>, from,<br />

http://www.ofta.gov.hk/datastat/eng_cus_isp.pdf.<br />

OFTA (Office of the Telecommunications Authority, Hong Kong Government) (2004b). The groundbreaking<br />

decade: major developments in the Hong Kong telecom industry in the past 10 years, retrieved March 14, <strong>2006</strong>,<br />

from, http://www.ofta.gov.hk/brochure/pdf/decade_eng.pdf.<br />

Rosenberg, M. J. (2001). E-Learning: Strategies for delivering knowledge in the digital age, New York, USA:<br />

McGraw-Hill.<br />

Ryan, S., Scott, B., Freeman, H., & Patel, D. (2000). The virtual university: The internet and resource-based<br />

learning, London: Kogan Page.<br />

Scott, P. (2001, November 13). Universities should resist e-imperialism. The Guardian, retrieved March 14,<br />

<strong>2006</strong>, from, http://education.guardian.co.uk/higher/news/story/0,9830,592147,00.html.<br />

Stokes, P. J., (2000). How e-learning will transform education. Education Week, 20 (2), retrieved March 14,<br />

<strong>2006</strong>, from, http://www.edweek.org/ew/ewstory.cfm?slug=02stokes.h20.<br />

Stoll, L., & Fink, D. (1996). Changing our schools: Linking school effectiveness and school improvement,<br />

Buckingham, UK: Open University Press.<br />

Trochim, W. M. (2002). Research Methods Knowledge Base (2 nd Ed.): Likert Scaling, retrieved March 14, <strong>2006</strong>,<br />

from, http://www.socialresearchmethods.net/kb/scallik.htm.<br />

UGC (University Grants Committee) (1996, <strong>October</strong>). Higher education in Hong Kong: A report by the<br />

University Grants Committee, Hong Kong: Hong Kong Government Printer.<br />

UGC (University Grants Committee) (2004). General statistics on higher education in Hong Kong: UGC-funded<br />

Institutions, retrieved March 14, <strong>2006</strong>, from, http://www.ugc.edu.hk/english/statistics/genstat.pdf.<br />

Williams, D. (2003, March 22). Success still in the distance. The Guardian, retrieved March 14, <strong>2006</strong>, from,<br />

http://education.guardian.co.uk/students/rise/story/0,13013,926678,00.html.<br />

Yeung, L. (2004). HKU marks out its global standing strategy. The South China Morning Post, September 18,<br />

2004, E3.<br />

Yeung, L. (2005). Self-reliance key to online English learning. The South China Morning Post, April 30, E3.<br />

94


Ateveh, K. & Lockemann, P. C. (<strong>2006</strong>). Reuse- and Aspect-Oriented Courseware Development. Educational<br />

Technology & Society, 9 (4), 95-113.<br />

Reuse- and Aspect-Oriented Courseware Development<br />

Khaldoun Ateyeh and Peter C. Lockemann<br />

Fakultät für Informatik, Universität Karlsruhe, Postfach 6980, 76128 Karlsruhe, Germany<br />

ateyeh@ipd.uka.de<br />

lockeman@ipd.uka.de<br />

ABSTRACT<br />

No longer can courseware providers deal with one homogeneous target group, one learning form and<br />

possibly one pedagogical approach. Instead they must develop a broad range of courseware, each serving its<br />

specific target group, each adjusted to a specific learning and teaching form, each appealing to its own<br />

learning and teaching scenario, and each incorporating its own pedagogical approach, and to do all this in a<br />

cost-effective and timely fashion. The thesis of this paper is that only an approach that is much more<br />

dictated by software engineering principles than what has been usual so far will meet these needs. Because<br />

of the economical constraints, the overriding engineering principle should be component reuse, and if<br />

several distinctive concerns become interwoven – above all content, didactics and technology – component<br />

reuse should be augmented by aspect-oriented programming. The paper develops and details a novel<br />

courseware engineering process that combines software reuse, component technology and aspect-oriented<br />

programming.<br />

Keywords<br />

Courseware engineering, Courseware reuse, Aspect-oriented development, Educational system development<br />

Introduction<br />

The goal of modern instructional design and learning theory are curricula that no longer follow a standardized<br />

pattern but are customized to the students' prior knowledge and learner type. But the goal also raises new<br />

challenges to courseware providers. No longer can they deal with one homogenous target group, one learning<br />

form and possibly one pedagogical approach. Instead they must develop a broad range of courseware, each<br />

serving its specific target group, each adjusted to a specific learning and teaching form, each appealing to its own<br />

learning and teaching scenario, and each incorporating its own pedagogical approach, and they must do all this in<br />

a cost-effective and timely fashion.<br />

This makes courseware development essentially an engineering challenge. It is a challenge, though, that has long<br />

been known to software engineering. There, component reuse is considered the key technique. Our vision, then,<br />

is to apply this technique to the creation of courseware that is highly adaptable and modular, and thus reusable in<br />

many contexts and for many needs. It should be possible to easily adapt the same courseware so that it can be<br />

used by students at universities and by employees in companies, by the student who prefers to learn online as<br />

well as by the one who prefers the traditional way of learning by printing out a script, by an instructor in the<br />

lecture hall or by one in a virtual class room. Stated differently, the way courseware components are to be reused<br />

depends on further factors such as pedagogical, psychological, and ergonomic aspects. Observing such factors<br />

during the development of component-base software has been known in software engineering as aspect-oriented<br />

programming.<br />

This paper demonstrates that despite the differences the engineering techniques of software reuse and aspectoriented<br />

programming can successfully and profitably be applied to courseware development, although they<br />

need to be specialized for the purpose. The paper is organized as follows. After examining the facets of reuse and<br />

aspect-orientation and the relevant state-of-the-art we develop the main ideas of a reuse-driven courseware<br />

development process. We introduce a model development process that allows us to separately pursue the<br />

different educational concerns in courseware. In particular we treat domain engineering and its specialization to<br />

content and didactics, and discuss the weaving of the concerns (or aspects) into a complete courseware. The final<br />

chapters cover the implementation and the conclusions.<br />

Software engineering for courseware<br />

The thesis of this paper is that courseware development can profit from certain principles of software<br />

engineering, specifically software reuse and aspect-oriented programming. We now take a closer look at the<br />

implications of these principles.<br />

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the<br />

copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies<br />

are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by<br />

others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior<br />

specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.<br />

95


Software reuse has two dimensions, building a stock of software components and combining components into a<br />

system with the desired properties. The first dimension is referred to as engineering for reuse and the second as<br />

engineering with reuse. Correspondingly in courseware development, engineering for reuse would refer to the<br />

design and development of learning assets (any reasonably delimited material which can contribute to a learning<br />

object) so that they can be reused later on when a specific teaching or learning environment is to be served, and<br />

engineering with reuse to synthesizing a particular courseware.<br />

Now, suppose a repository of learning assets that has been built up over time. Imagine a teacher or learner who<br />

needs instructional material on a certain subject. In all likelihood he/she will inspect the repository for assets that<br />

cover the desired content. Suppose a suitable asset has been found. Then the particular circumstances of the<br />

teacher or student come into play. For one, these determine the didactics to be pursued, that is, the content<br />

should be organized to account for the expected background and experience, the desired learning form and<br />

scenario, and the pedagogical approach. For another, the circumstances dictate how the material is to be accessed<br />

and – above all – how it is to be technically presented, e.g., visually by static or animated graphics, video<br />

sequences, audio, or several in combination, and what technical arrangement to choose. Content, didactics and<br />

technology constitute what we call aspects. Courseware development must observe these aspects. It is the<br />

combination of component reuse and aspect observation that seems to pose major particular challenges.<br />

Existing contributions to the problem<br />

Software engineering contributions<br />

The literature distinguishes between two main categories of software reuse approaches, component-based reuse<br />

and generation-based reuse (Biggerstaff & Perlis, 1989; Henninger, 1997; Szyperski, 2003a). A componentbased<br />

development process describes all activities in the context of a complete software life cycle on the basis of<br />

components. A software component has contractually specified interfaces and explicit context dependencies, can<br />

be deployed independently, and is subject to composition by third parties (Szyperski, 2003b). This matches our<br />

goal of deploying entire learning objects and configuring them into larger courses, following, e.g., the SEI<br />

reference model (SEI, 2004 ).<br />

Component-based reuse, or more precisely systematic, reuse-based development of system families in a domain<br />

rather than one-of-a-kind systems is the objective of Domain Engineering (DE). As defined in AOSD (2003),<br />

domain engineering is the activity of collecting, organizing and storing past experience in building systems or<br />

parts of systems in a particular domain in the form of reusable assets, as well as providing an adequate means for<br />

reusing these assets when building new systems. What a domain is in a specific situation is left to the consensus<br />

of the so-called stakeholders in that domain. The practical goal of the domain engineering process is to build a<br />

reference architecture that can be easily reused across members of a system family or families in that domain.<br />

Domain engineering appears particularly attractive to us, since its objective seems to concur with the engineering<br />

for reuse phase.<br />

Generation-based reuse takes a higher-level specification of a software artifact and produces its implementation<br />

via a generator. The users of a generator see a system that allows them to go from a specification to a software<br />

component without having to understand the internal details of the generator (Bell et al., 1994). A specific<br />

representative of generation-based reuse, Aspect-Oriented Programming (AOP), seems particularly suited to our<br />

problem of separating and integrating aspects. AOP deals with separation of concerns at the implementation<br />

level and tries to provide linguistic mechanisms to factor out different aspects of a program, which can be<br />

defined, understood, and evolved separately (Czarnecki & Eisenecker, 2000). The goal is to provide methods and<br />

techniques for decomposing problems into a number of functional components as well as a number of aspects<br />

that cut across functional components, and then compose these components and aspects to obtain a final system<br />

implementation (IBM, 2004). Unfortunately, the definition sounds very abstract, and even the three steps of<br />

aspectual decomposition, concern implementation and aspectual re-composition by weaving remain abstract<br />

principles. Not surprisingly, then, AOP is more an open research agenda than an existing technology that one can<br />

readily use. First approaches are subject-oriented programming (SOP) (IBM, 2004), Adaptive Programming<br />

(AP) (Lieberherr, 1996), and Composition Filters (CF) (Aksit, 1989), all of them tailored to object-oriented<br />

programming. So while the concepts sound attractive to our problem, we would need to invest into further<br />

research.<br />

96


Systematic courseware development<br />

Traditional methods of courseware development have a tendency towards courseware for specific learning<br />

situations, learner characteristics, learning objectives, or learning/teaching strategies. Consequently, there has<br />

been scant need for systems that are adaptable to varying needs. Rather the result is a monolithic structure that<br />

mixes different aspects such as content, didactic, and technical aspects in inseparable ways, and where one of<br />

these aspects dominates the others.<br />

Take the didactic-oriented model that concentrates on the didactic aspect of courseware development. Most<br />

prominent is the instructional design (ID) model that tends towards the design and development of so-called<br />

tutorial and drill-and-practice systems (see, e.g., Tennyson & Rasch, 1995; Merrill et al., 1990). While<br />

considered outdated by today‘s learning/teaching methods objectives, the model still is interesting because of its<br />

strength in its methods for analyzing learning situations, characteristics of the learners, learning objectives, and<br />

for designing suitable learning/teaching strategies. Or as another example, take technology-oriented models that<br />

focus on the technical and engineering aspects of courseware under the assumption that the use of multimedia<br />

and/or hypermedia as well as technical tools will significantly enhance the learning/teaching process. Typical<br />

examples can be found in (Boles & Schlattmann, 1998; Garzooto et al.,1991; Isakowitz et al., 1995) or as<br />

commercial products (Macromedia, 2004a; Macromedia, 2004b; ToolBook, 2004). What these models have to<br />

offer to other approaches, though, are the sophisticated authoring environments and a structured development<br />

process divided into phases of information objects production, authoring, and generation. Courseware<br />

engineering models seek to combine the strengths of the two other models. They consider courseware<br />

development as a mixture of software development along the line of a process of developing business software,<br />

and instructional design as the process of developing instructional or didactic models. However, the main focus<br />

of these models is to provide a systematic development process that allows one to manage the growing<br />

complexity of courseware development. Little or no consideration is given to courseware reuse. Examples are<br />

the Essener-Learn-Model (Pawlowski, 2001) and the IntView Lifecycle Model (Grützner et al., 2002) as well<br />

some recent research results (Blumstengel, 1998; Klein, 2002).<br />

Even a focused, monolithic system can find many users provided it is accessible from outside places. This has<br />

been recognized over the past years by the eLearning community which has come up with various standards in<br />

order to ensure reuse and interoperability. Relevant standards are the Learning Object Metadata (LOM) standard<br />

(IEEE, 2003), the IMS Content Packaging (IMSCP) standard (IMS, 2004), the IMS Learning Design standard<br />

(IMSLD, 2004), the Learning Technology Systems Architecture (LTSA) standard (IEEE, 2001), and the<br />

Sharable Content Object Reference Model (SCORM) (ADL, 2004). The standards cover nearly all aspects of<br />

learning such as didactic, content, learner profile, learning management system, and their use seems essential for<br />

the support of reusability at an inter-community/inter-organization level and of interoperability among<br />

courseware/learning management systems.<br />

Clearly, any effort towards more flexible courseware development should take cognizance of these<br />

standardization efforts even though their large number certainly is confusing. In addition, they make assumptions<br />

on the development process that do not seem entirely realistic. One assumption is that a component can be<br />

reused in new contexts just as it is. This seems unrealistic for the content and didactic aspects: In our experience<br />

these seem highly interdependent, so that one cannot simply transfer a component into a different didactic<br />

environment without adapting its content. Under IMSLD, even the instructional or didactic models are designed<br />

with a specific learning context in mind so all one can do is reuse the corresponding components if their context<br />

happens match the selected or designed instructional strategy.<br />

There has been some work on generating courseware from components, given they fit the requirements. Merrill<br />

introduces the concept of instructional transaction shell to organize existing so-called instructional transactions<br />

and knowledge objects into new courseware (Merrill et al., 1990). On the other hand, little is being said about the<br />

process of the development of such reusable elements, rather the focus is on the process of automatic generation<br />

of instructions from existing reusable instructional transactions and knowledge objects.<br />

In conclusion, we face a number of challenges. It should be possible to design courseware in small units, with a<br />

clear separation of the various aspects, in our case predominantly contents and didactics. Their interdependence<br />

should be taken into account when courseware is produced for a specific learning/teaching context.<br />

Consequently, there should be a clear separation of the modeling phase – corresponding to engineering for reuse<br />

– and the production phase – corresponding to engineering with reuse. The production phase should largely be<br />

automated, somewhat in the sense of model-driven programming.<br />

97


Courseware development process model<br />

The challenges as discussed in the previous section seem to suggest a crude process model consisting of two<br />

major processes, engineering for reuse and engineering with reuse. Figure 1 gives an overview. The two<br />

processes are referred to as domain engineering and course engineering. Courseware engineering must start with<br />

domain engineering because it is this process that lays the foundation for course families and, hence, for a<br />

variety of courseware even for the same topic, where the differences reflect the situation to be served. Only when<br />

a family has been established – at least in part – can individual family members be developed. In practice,<br />

though, the processes are not strictly separated but iteratively performed. For example, if we assume a repository<br />

to hold all developed assets the repository may not entirely satisfy all current needs so that one may decide to<br />

ignore an asset and build a new, suitable one from scratch. The result will be stored in the repository and thus<br />

become part of some other development.<br />

Both processes are themselves iterative processes. The domain engineering process is relatively independent of<br />

the course engineering process whereas the course engineering process strongly depends on the results of the<br />

domain engineering process. Therefore, there is only limited potential for conducting the two in parallel. On the<br />

other hand, new requirements that have a general character and have not yet been considered in the domain<br />

engineering process but are first discovered in the course engineering process are fed back to the domain<br />

engineering process.<br />

Figure 1. An overview of the development process<br />

Both processes are further divided into phases. The division according to Figure 1 is only effective if, beyond<br />

associating clear milestones with each phase and allowing for collective development by the various experts, one<br />

can clearly identify which phases from domain engineering contribute which results to which phases of the<br />

course engineering process. The answer to this problem is the subject of the remaining chapters.<br />

How do the software engineering techniques we plan to employ contribute to the courseware development<br />

process?<br />

Domain engineering is geared towards the development of families within a given knowledge domain.<br />

Aspect-oriented programming allows one to concentrate on different aspects one at a time (a principle often<br />

also referred to as separation of concerns).<br />

Component technology allows one to assemble the learning assets into larger but still self-contained courses.<br />

Domain engineering is the general principle that underlies our process model. Aspect-oriented programming<br />

gives substance to the principle and will affect all three phases of the domain engineering process. We consider<br />

three aspects:<br />

The contents of the course (content aspect): The instructor needs to decide on a syllabus of the course,<br />

choosing which (sub-)topics to include, how much emphasis to put on each, and deciding on an order of<br />

presentation.<br />

The didactic strategy and methods to use (didactic aspect): The instructor needs to decide which didactic<br />

strategy is suited best to reach the objectives of the course.<br />

98


The learning/teaching context or constraints under which the course will be taught (technical aspect): These<br />

include the number of participants, the amount of time available, the room and the technical equipment that<br />

will be used, whether the course will be face-to-face or distance learning, whether teaching will be<br />

synchronous or asynchronous and so on. We will not discuss the technical aspect in detail but include it in<br />

the didactic aspect whenever appropriate.<br />

Figure 2. Influence of existing techniques on the development process<br />

Finally, component technology is the basis for constructing the learning objects for a specific course from the<br />

assets and should follow one or more of the established techniques for constructing components.<br />

Figure 2 summarizes how the three techniques impact the domain engineering process and where, according to<br />

the earlier discussion on systematic courseware development, instructional design and standards exert an<br />

influence.<br />

The domain engineering process<br />

The gross structure of domain engineering consists of the three phases of domain analysis, domain design, and<br />

domain implementation. All three are mainly dominated by considerations of the domain knowledge – the<br />

contents – and of three independent aspects – content, didactics, and technology. Below we study the effects<br />

mainly of the first two factors on each phase. A course on database systems will illustrate the development<br />

process.<br />

Domain analysis<br />

Content analysis<br />

During content analysis the scope of the knowledge domain is selected and defined. It is the responsibility of a<br />

larger community to agree on the general topics that should be covered by the domain model as well as those to<br />

be left out. Figure 3 gives some examples. The main contributions come from material used by the community<br />

members in conducting their own courses, from reference textbooks, from knowledge domain experts in the<br />

community, and perhaps from available knowledge domain ontologies. As part of the discussion one should<br />

refine the general concepts into more specific ones, assign terms to them and collect these terms in a vocabulary.<br />

99


Didactics analysis<br />

This phase considers both, content didactics and process didactics. The latter covers general properties such as<br />

teaching/learning goals and contexts, potential audiences. The former focuses on the definition and selection of<br />

the various didactic principles through which the learning materials will be submitted to the students. Figure 3<br />

shows some examples.<br />

Obviously, didactic principles are generic, i.e., independent of a specific knowledge domain. Consequently, they<br />

can be specified just once and early on, and can then be applied to a broad range of knowledge domains.<br />

Domain design<br />

Figure 3. Sample result of the domain analysis<br />

The purpose of the domain design phase is to develop a more formal courseware model from which one can<br />

systematically derive an implementation.<br />

Content design<br />

Content design defines a shared content model for the content concern. The content model aims at the creation of<br />

a shared understanding for the knowledge domain that is of interest to the members of the community<br />

independent of its use in a specific learning context (Ateyeh et al., 2003). Basically one determines the<br />

conceptual entities that form a common basis for the reuse and exchange of courseware content, where each unit<br />

can be (re)used by each member of the community.<br />

Ontologies appear well-suited for the purpose. An ontology is an explicit specification of a conceptualization,<br />

where a conceptualization is an abstract, simplified view of the world that we wish to represent for some<br />

purpose. The purpose of an ontology is the shared understanding of some domain of interest (Uschold &<br />

Gruninger, 1996). It usually takes the form of a set of concepts (e.g. entities, attributes, processes, etc.), their<br />

definitions, and their relationships.<br />

The basis of a knowledge domain ontology are the concepts identified during analysis. These are organized into<br />

a relational structure, with the relationships dictated by the purpose of courseware. Two types of relationships<br />

seem sufficient:<br />

isSubTopicOf: Two terms A and B have relationship B isSubTopicOf A if A is more generic than B. E.g.,<br />

“relational algebra” is more generic than “join operator”: join operator isSubTopicOf relational algebra.<br />

isPrerequisiteFor: Two terms A and B have relationship A isPrerequisiteFor B when the knowledge of A is<br />

needed to understand B. In our example, knowledge about functional dependencies is needed before tackling<br />

normalization: functional dependency isPrerequisiteFor normalization.<br />

100


Figure 4. Sample domain knowledge ontology (White arrow: isSubTopicOf, Black arrows: isPrerequisiteFor,<br />

both relationships are transitive)<br />

Designing an ontology is by no means trivial, nor are there commonly accepted standard methods for building<br />

ontologies. We refer the reader to the work of Holsapple and Joshi (2002) on collaborative ontology design.<br />

Figure 4 shows a very small part of an ontology for the domain ''data base systems'' that captures the topic of<br />

normalization. The resulting ontology can then be implemented using one of the many ontology representation<br />

languages such as RDF/RDFS, OWL, DAML, KIF, etc. The different representation languages provide different<br />

levels of formality to capture different kinds of ontology characteristics and semantics. We found RDF to be<br />

sufficient four our purpose.<br />

We note that content analysis and content design have to be done just once for each knowledge domain. Ideally,<br />

one could then defer the combination of ontology and didactic principles to the domain implementation phase.<br />

Content didactics design<br />

As pointed out earlier, content and didactics are interdependent so that one cannot simply transfer a component<br />

into a different didactic environment without adapting its content. Indeed, we found it extremely difficult to<br />

delay their combination past content didactics design. Instead we merge the ontology with the didactic<br />

principles, i.e., the intended didactic usage of the content, into what we call learning atoms. Figure 5 gives a few<br />

examples. Figure 5 also demonstrates that atom types may belong to terms on any level of the ontology.<br />

Learning atoms that belong to the same term in the ontology are combined into a learning module. Hence, for<br />

each term in the ontology there exists exactly one corresponding learning module with all learning atoms related<br />

to that term. As we move upwards to higher levels of the ontology, more complex modules are constructed that<br />

include modules from the lower levels. Consequently, the notion of learning module is recursively defined.<br />

Figure 5 sketches an example.<br />

Since there is a learning module for each term in the ontology, modules are already implicitly defined during the<br />

creation of the ontology. By instantiating an atom from a certain atom type it automatically becomes part of the<br />

appropriate module (and implicitly also of the higher-up modules through the isSubtopicOf relationship).<br />

101


Process didactics design<br />

We focus on two aspects of the so-called learning process didactics:<br />

The Learning/teaching strategy: Structures the learning/teaching process into one or more phases. Each<br />

phase meets a specific didactic goal. For every learning/teaching phase a suitable didactic method is<br />

used.<br />

The Didactic method: Is a system of didactic rules that instructs or guides students or teachers while<br />

learning or teaching. A didactic method achieves certain learning or teaching goals under given<br />

circumstances.<br />

As an example for a learning/teaching strategy take the problem-based learning/teaching strategy of Merrill<br />

rooted in the constructivism learning theory (Merrill, 2000). A second example and the one we use to illustrate<br />

our approach is the three-step learning/teaching strategy of Figure 6 with three main phases:<br />

Introduction phase: It defines the starting point and the goals of the learning process.<br />

Working phase: This phase consists of further sub-phases (didactic functions): setup, workthrough,<br />

apply, transfer, assess and integrate, and should be supported by a suitably stimulating environment.<br />

Completion phase: The goals of the learning process must be secured.<br />

Independent of the didactic functions one can define didactic methods. These are shown in Figure 6 below the<br />

solid line to indicate that, at least in principle, each may be associated with each didactic function.<br />

In the literature, learning/teaching strategies and didactic methods are mostly (if at all) expressed in natural<br />

language. Clearly this is not suitable in a cooperative environment that tries to increase courseware reuse and<br />

that should be based on automatic processing of courseware entities. What is needed is some kind of formalism<br />

that allows to capture the didactic concern and to easily adapt them to a given learning/teaching context. Figure 7<br />

graphically illustrates our own didactic metamodel. The figure also indicates (shaded box) where the didactic<br />

methods are connected to the didactic functions. Further, the figure demonstrates how to integrate the technical<br />

aspect with the didactic aspect. The implementation has been done in XML.<br />

Figure 5. Sample learning modules<br />

102


Figure 6. Classical three-step learning/teaching strategy and associated didactic methods<br />

Domain implementation<br />

Figure 7. The didactic method metamodel<br />

In this phase concrete reusable courseware entities (learning assets) for the models defined in the design phases<br />

(content entities, didactic entities, software tools) are implemented and then kept in a repository for subsequent<br />

use in the course engineering process. They serve as a starting kit for the courseware development in the course<br />

engineering process.<br />

Content implementation<br />

Strictly speaking, what we defined in the design phase are learning atom types and module types. Atom types are<br />

instantiated to one or more learning atoms: The content is developed in detail and perhaps with different foci<br />

depending on the background of the learner and the intentions of the teacher, representations are chosen such as<br />

powerpoint slides or pdf files, and both are combined into an executable unit in a form that can be reused in<br />

different environments and situations (for an example, see Figure 8). The atoms are stored in a content<br />

repository.<br />

103


The repository is organized into containers. A container maintains the contents of a learning module.<br />

Consequently, there exists a container for each term in the ontology. At a minimum, for a lowest-level term a<br />

container consists solely of learning atoms. In general, a container may hold atoms as well as “smaller”<br />

containers for the less generic modules. Taking the example of Figure 5, there are containers for functional<br />

dependency (with atoms only), dependencies (it would include the functional dependency container),<br />

normalization, etc.<br />

Didactics implementation<br />

In a sense, domain design “atomizes” the learning material into a combination of local contents and local<br />

didactics. However, learners are to be offered self-contained courses. A course is subject to a content didactic<br />

strategy. Such a strategy starts from a didactic function, determines the associated didactic method and defines in<br />

the form of a procedure (Figure 7) the didactic principles through which to reach the students, and is represented<br />

by a so-called didactic template. The central idea of our approach is to include in the repository a number of such<br />

templates as assort of preplanning the potentially desired course structures. Figure 9 gives an example for a<br />

didactic template that could be used for a presentation during the setup phase. Suppose we apply it to an SQL<br />

course module. The template states that the course should begin with an overview followed by a motivation and<br />

an activation atom. Subsequently, for each sub-topic of the course an explanation or a definition of the topic is<br />

needed followed by an example. Finally the course should finish with a concluding example.<br />

Since our goal is to automatically generate a course from the content repository and a given template, templates<br />

should be formulated in a machine-processable script language. For the purpose, we developed DidaScript, a<br />

typical script language with the usual constructs for variable declarations, assignment statements, conditional<br />

and loop expressions. DidaScript includes built-in functions for the navigation in a course structure of learning<br />

modules, for ordering all or selected atoms of a specified module, and for hiding atoms in a module (Ateyeh,<br />

2004). The resulting course structure is represented via XML.<br />

Figure 8. Sample learning atoms<br />

104


Figure 9. The classical three-step learning/teaching strategy augmented by didactic methods and the association<br />

of a content didactic template<br />

The course engineering process<br />

Figure 10. A sample course specification<br />

Domain engineering is engineering for reuse and results in a host of learning assets and organizes them in an<br />

organizational structure. Course engineering is engineering with reuse and employs the results of the domain<br />

engineering process to construct from them, with moderate effort, context-specific courses in the form of<br />

learning objects (We use the term ''course'' as a general notion that refers to any learning/teaching event such as<br />

course, seminar, lesson, etc.).<br />

We pursue two goals. One is to fuse the outputs of content engineering and didactics engineering (or in AOP<br />

jargon, to weave them). The second goal is to do the fusion as much as possible in an automated fashion, by<br />

drawing on the repository.<br />

Requirements analysis<br />

The goal of this phase is to analyze and define the (general) requirements for the course, such as the<br />

learning/teaching topics, target group, objectives, learning form, time frame etc. In this phase both results from<br />

105


the analysis and the design phase of the domain engineering process are used. The requirements are expressed<br />

through so-called general specifications for the learning/teaching event (see Figure 10 for an example).<br />

Course configuration<br />

Didactics configuration<br />

Based on our initial experience we recommend to start the course configuration with didactics. As a first step the<br />

requirements are re-formulated as a learning strategy. Consider nested queries as a subtopic of an SQL course.<br />

Using the classical three-step learning/teaching strategy we proceed as follows: In the introduction phase,<br />

existing knowledge about SQL should be recapitulated, and it should be motivated why nested queries are<br />

needed. In the working phase, the syntax of nested queries should be introduced (set-up), some examples should<br />

be shown (work-through and apply), given natural language requests for specific information nested queries<br />

should be formulated (transfer), nested queries should be compared to alternative ways to obtain the same<br />

information (assess) and nested queries should be used together with other SQL constructs (integrate).<br />

In the next step the learning strategy must be translated into an assignment of didactic methods. To return to our<br />

example of nested queries, the introduction phase could require students to try and formulate queries with the<br />

SQL constructs they already know, revealing the need for nested queries (active structuring). Then, a<br />

presentation of the syntax of nested queries followed by the online presentation of some examples could cover<br />

the set up and apply phases, the students could then individually formulate nested queries to gain information, a<br />

collaborative discussion comparing nested queries to alternative formulations would be an appropriate method<br />

for the assessing phase. A computer session, requiring the usage of nested queries together with other SQL<br />

constructs could serve both to integrate and to secure the knowledge gained (Figure 9 is a good illustration).<br />

Given these assignments, we choose an appropriate – hopefully predefined – didactic template from the<br />

repository. The template in turn defines the didactic principles to be applied to the material. Together with the<br />

content structure we now know which learning atoms and modules to select from the repository.<br />

Content configuration<br />

During content configuration we have to decide on the precise content (learning modules). The process of<br />

didactics configuration gives numerous suggestions on how to organize the content. Therefore, didactic and<br />

content configuration run somewhat in parallel. Remember that a learning module contains all types of material<br />

from all participants that can be associated with the corresponding term in the ontology. Consequently, the<br />

course that evolves from choosing the topics (modules) from the ontology reflects the structure predefined by the<br />

isSubtopicOf relationship of the domain-specific ontology. The instructor can now adapt this structure to his<br />

requirements. In particular, she/he selects from the predefined structure those sub-topics (modules) she/he<br />

intends to address in her/his course, and defines structural relationships between the selected topics which define<br />

the navigation structure that has to be used by the students and/or the instructor. The development platform may<br />

support her/him by warning her/him to avoid a structure that would violate the relation isPrerequisiteFor defined<br />

in the domain-specific ontology. To continue our earlier example, we would obtain for the part of the course that<br />

deals with normalization the sequential structure of Figure 11. Comparing it with Figure 4 we note that all direct<br />

sub-modules of normalization were selected, whereas for module normal forms the sub-module BCNF has been<br />

deleted.<br />

Weaving a course<br />

Given the template and the course structure, we can mechanically draw the necessary learning atoms and<br />

modules from the repository and weave them into a course flow. Automation is in the form of a generator.<br />

Concrete learning atoms can be drawn from the repository by the generator merging, in a stepwise fashion, the<br />

learning atom type from the content structure and the didactic principle from the didactic template. Figure 12<br />

illustrates the principle for our normalization example. The course generator selects from the root learning<br />

module only those learning atoms that follow either the didactic principle of overview or motivation. These are<br />

then ordered so that the overview atoms precede the motivation atoms. Further, the strategy states that for each<br />

sub-module of the root module an explanation or a definition of the topic should be given followed by an<br />

106


exercise or an example. Finally the course generator visits the root learning module again to select a concluding<br />

example followed by an exercise and the solution for that exercise.<br />

Assembly, test and integration<br />

Figure 11. Sample course structure<br />

The course modules resulting from the configuration phase are normally half-finished modules. E.g., for some<br />

learning topics appropriate learning atoms may not be found in the repository. In this case, an appropriate<br />

learning atom must be developed and added to the repository for future use. In this case, the instructor switches<br />

back into his role as an author, develops the appropriate material and adds it to the repository. Furthermore, in<br />

many cases it is necessary to add a transitional content between the different learning atoms as well as contextspecific<br />

atoms such as an overview, a motivation, and a summary for the whole course. In many cases it is also<br />

necessary to adapt the layout of some of the learning atoms to meet a selected standard layout for the whole<br />

course.<br />

We are now in a position to complete Figure 2 (Figure 13).<br />

Proof of concept<br />

Project background<br />

The work underlying the paper evolved as part of the five-year program ViKar (Virtual University Karlsruhe)<br />

that, among others, undertook to explore how to develop shared course material that could subsequently be<br />

adapted to the needs of various schools of higher learning with different educational goals. Our focus was less on<br />

content and more on methodical issues of how to prepare the individual courses. Hence, we simply used an<br />

already existing suite of one-semester database courses that were of interest to all participating schools. The<br />

student sample was relatively broad: Third-year computer science students of a university, second-year business<br />

school students of a university, third-year students of a polytechnical school, and third-year students who<br />

alternated between the school and work at industry. Consequently, there were considerable differences with<br />

regard to both the selection and depth of the teaching material, and the didactic strategies. Our approach evolved<br />

slowly as we tried to deal with the – sometimes sobering – experiences, often as mundane as differing notations<br />

and terminology, or incompatible examples for illustration and exercises.<br />

107


Figure 12. Weaving content and didactics: Example<br />

Figure 13. Figure 2 refined<br />

108


Implementation<br />

Much of the effort went into domain engineering. Over the years a complete ontology for database technology<br />

was developed, driven both by standard textbooks and the powerpoint presentation of a complete 16-week<br />

university course at 3 hours per week, augmented by course material from the other three schools. Once a<br />

significant portion of the ontology was available we were able to systematically determine a large set of learning<br />

objects (atoms and modules) for the university courses and selective sets for the other schools, and complete<br />

them using much of the existing material.<br />

We could thus demonstrate that working from an ontology is indeed a viable approach to domain engineering. In<br />

particular, the approach is very well-structured and gives conciseness and unambiguity to the notion of learning<br />

object. We are certain that the same approach can be applied to all disciplines with a well-structured domain,<br />

such as the natural and engineering sciences.<br />

Technical platform: SCORE<br />

To support the experiments we developed a prototypical learning system, SCORE (System for COurseware<br />

REuse) (SCORE, 2004). SCORE distinguished between different participants of the learning/teaching process<br />

such as author, instructor, and student, and provides a suitable environment for each of these groups.<br />

SCORE is based on standard technologies that support openness, portability, and reusability, particularly<br />

relational/XML databases (Oracle9.1), Java, J2EE and XML, learning technology standards such as LOM and<br />

IMS content packaging, and design patterns such as Three-Tier architecture and Model-View-Controller. Figure<br />

14 shows the system architecture of SCORE with three layers:<br />

Figure 14. SCORE architecture<br />

1. The data base and data access layer (layer 1) manages all types of data needed or produced during the<br />

processes of courseware development and reuse, learning, and teaching. In particular, this layer includes a<br />

courseware repository that manages the different kinds of courseware entities such as learning objects<br />

(learning atoms, modules, course module) and didactic entities (didactic methods, learning/teaching<br />

strategies, didactic templates).<br />

2. The basic services layer (Layer 2): As the core of the SCORE system the layer provides a modular<br />

framework of common tools and services that are necessary for conducting different activities or tasks by<br />

the different user types. These tools or services are reusable building blocks used to assemble or build<br />

109


complete user-specific (author, instructor, and student) environments at layer 3 (see below). The services or<br />

tools found in this layer can be classified into three categories:<br />

Courseware development tools. These tools are used in the course engineering process and help the<br />

developer to conduct the different activities of the development process in an efficient way. Examples<br />

for tools in this category are AtomBuilder, ModuleBuilder, DidacticBuilder, and CourseBuilder.<br />

Learning/teaching tools. These tools are used in the learning/teaching process. From the course<br />

engineering point of view, these tools are considered reusable technology components that can be<br />

combined to provide a suitable learning/teaching environment for a specific learning/teaching method<br />

and content. Examples for tools in this category are CoursePresenter, Annotation, Chat/Mail.<br />

General tools. These are tools that are used by the different user types. Examples for such tools are the<br />

OntoBrowser and ScoreSearch.<br />

3. The user environments layer (layer 3): It includes specialized environments for each user type (author,<br />

instructor, and student). We restrict our discussion to the author environment as the one most relevant to this<br />

paper. The environment provides tools that allow one to effectively conduct the courseware development<br />

process. The components are integrated within a tool we call ''Course Wizard (CW)''.<br />

The AtomBuilder is used to build a new learning atom. It provides a comfortable user interface for<br />

describing a learning atom by SCORE metatadata and assigning it to a learning topic (module) in the<br />

domain-specific ontology. For the production of the atoms themselves, the author is able to use any<br />

available content production tools.<br />

The ModuleBuilder supports the course (content) engineering process. It is mainly used to deal with<br />

learning modules and provides all functionalities necessary for transforming a (general) learning<br />

module into a (learning context specific) course module. Therefore, the ModuleBuilder makes use of<br />

other tools such as the OntoBrowser and ScoreSearch.<br />

The DidacticBuilder deals with the learning process didactic discussed. It is used to find, select, create,<br />

and reuse learning/teaching strategies and methods.<br />

The CourseBuilder allows to build complete learning/teaching events (course, seminar, …). It is used to<br />

merge and integrate the different components of the different concerns to a complete learning/teaching<br />

event.<br />

The TestBuilder supports the creation of tests for the students.<br />

Figure 15 Course Wizard: Screenshot<br />

Figure 15 shows a screen shot of the CW with, from right to left, the ModuleBuilder, the DidacticBuilder and the<br />

CourseBuilder. The screen shot nicely shows how the user interface reflects the aspect separation of content and<br />

didactic. A complete example of how to use the CW in practice can be found in (Ateyeh, 2004).<br />

110


Evaluation and conclusions<br />

After reviewing the state of the art, we set ourselves several goals. It should be possible to design courseware in<br />

small units, with a clear separation of the various aspects, in our case predominantly contents and didactics.<br />

Their interdependence should only be taken into account when courseware is produced for a specific<br />

learning/teaching context. There should be a clear separation of the modeling phase – corresponding to<br />

engineering for reuse – and the production phase – corresponding to engineering with reuse. The production<br />

phase should largely be automated, somewhat in the sense of model-driven programming. Our approach was<br />

modeled after the software engineering techniques of software reuse, component technology, and aspect-oriented<br />

programming.<br />

The combination of these techniques poses challenges even under ordinary circumstances and has never before<br />

been applied to courseware engineering. As a first attempt in this direction we feel we have been quite<br />

successful. Content is the driving force. Through a domain engineering approach that is based on a commonly<br />

agreed ontology a first component structure is defined. Component adaptability is achieved by introducing the<br />

concept of learning resource type. It allows refining an ontology-based component in a large variety of ways that<br />

each suit a specific instructional intent. The resulting so-called learning atoms can then be implemented and<br />

maintained in a repository.<br />

We also managed to separate the aspects of contents and didactics. We could show that one can derive a host of<br />

didactical methods for a given didactical strategy, and then detail the method in the form of a template that<br />

prescribes how atoms from the repository should be composed and ordered to meet the instructor’s and/or<br />

student’s objectives. A specific course can be generated almost automatically by a weaving process that takes a<br />

template as a prescription for which atoms to draw from the repository and how to arrange them in order.<br />

Nonetheless, the results are only a first and still modest step towards the flexible and economical production of<br />

courseware for a large variety of settings. One weakness of our work is the dearth of empirical evidence on the<br />

suitability of our approach. The approach evolved slowly in the course of a five-year project on a virtual<br />

university. Originally the emphasis was on the content level by ontology engineering. During the course of the<br />

work it became apparent that the desired flexibility could only be achieved if content and didactic were dealt<br />

with separately. This resulted in the concepts of didactic template and aspect weaving by generation.<br />

Unfortunately, the short remaining time of the program did not allow us to test the strategy on a larger scale.<br />

As a consequence, more empirical evidence must be gathered to convince oneself that the separation of content<br />

and didactic works under many conditions, and what these conditions are. Likewise, before undergoing the vast<br />

effort of building an ontology criteria would have to be developed whether a subject area is well-structured<br />

enough to justify the effort. As a first step, one should apply the approach to additional domains beyond the<br />

subject area of database courses. Also, for reasons of economy, it seems incumbent to integrate the SCORE<br />

courseware development and reuse environment into existing commercial learning platforms that support the<br />

learning technology standards and provide (to some extent) a modular learning/teaching environment.<br />

We were not entirely successful to separate the aspects of contents and didactics. Although we did better than<br />

other current work, we still had to merge content and didactic principles – as judged from our goals –<br />

prematurely during the domain design phase. Future work should concentrate on how to shift more of the<br />

merging into course engineering.<br />

Another characteristic of the approach is its heavy reliance on the existence of a suitable ontology. At least<br />

initially ontologies will have to be developed quite often, and this requires a fairly large investment in time and<br />

competent manpower and, therefore, there are considerable delays before the first course becomes available for a<br />

particular subject area. Consequently, one should steadily observe the market for ontology development tools.<br />

Acknowledgment<br />

The authors gratefully acknowledge the collaborative and stimulating environments of ViKar with Wolffried<br />

Stucky, Klaus Gremminger, Rudolf Krieger, Daniel Sommer, Jutta Mülle, and of SCORE with Birgitta König-<br />

Ries and Michael Klein.<br />

111


References<br />

ADL (Advanced Distributed Learning Initiative) (2004). Sharable Content Object Reference Model (SCORM):<br />

SCORM 2004 3 rd Edition, retrieved May, 30, <strong>2006</strong> from http://www.adlnet.gov/scorm/downloads/index.cfm.<br />

AOSD (2003). Aspect-Oriented Software development, retrieved May, 30, <strong>2006</strong> from http://aosd.net.<br />

Aksit, M. (1989). On the Design of the Object-Oriented Programming Language Sina. Ph.D. thesis, University of Twente.<br />

Ateyeh, K. & Klein, M. & König-Ries, B. & Mülle, J. (2003). A Practical Strategy the Modularization of Courseware<br />

Design. In: Professionelles Wissensmanagement – Erfahrungen und Visionen: Adaptive E-Learning and Metadata, Luzern.<br />

Ateyeh, K. (2004). Reuse-Driven Courseware Engineering. Ph.D. thesis, Universität Karlsruhe. Shaker Verlag.<br />

Bell, J., Bellegarde, F., Hook, J., Kieburtz, R. B., Kotov, A., Lewis, J., McKinney, L., Oliva, D. P., Sheard, T., Tong, L.,<br />

Walton, L.& Zhou, T. (1994). Software Design for Reliability and Reuse: A Proof-of-Concept Demonstration. In<br />

Proceedings of the Conference TRI-Ada.<br />

Biggerstaff, T. J. & Perlis, A. J. (1989). Software Reusability, Concepts, and Models. Vol. 1. ACM Press. Addison-Wesley.<br />

Blumstengel, A. (1998). Entwicklung hypermedialer Lernsysteme. (Development of Hypermedia Learning Systems, in<br />

German) Ph.D. thesis, Universität Paderborn.<br />

Boles, D. & Schlattmann, M. (1998). Multimedia-Autorensysteme: Grafisch-interaktive Werkzeuge zur Erstellung<br />

multimedialer Anwendungen. (Multimedia Author Systems: Graphical-interactive Tools for the Development of Multimedia<br />

Applications, in German) LOG IN 18 (1), 10-18.<br />

Czarnecki, K. & Eisenecker, U. W. (2000). Generative Programming. Addison-Wesley.<br />

Garzotto, P., Paolini, P. & Schwabe, D. (1991). HDM – a Model for the Design of Hypertext Applications. In Proceedings of<br />

the 3rd Annual ACM Conference on Hypertext Systems, 313-328.<br />

Grützner, I., Pfahl, D. & Ruhe, G. (2002). Systematic Courseware Development Using an Integrated Engineering Style<br />

Method. In: Networked Learning NL2002, Berlin.<br />

Henninger, S. (1997). An Evolutionary Approach to Constructing Effective Software Reuse Repositories. ACM Trans.<br />

Software Engineering and Methodology (TOSEM) 6 (2), 111-140.<br />

Holsapple, C. W. & Joshi, K. D. (2002). A Collaborative Approach Ontology Design. Comm. ACM 45 (2).<br />

IBM (2004). Subject-Oriented Programming, retrieved May, 11, <strong>2006</strong> from http://www.research.ibm.com/sop/.<br />

IEEE Learning Technology Standards Committee (LTSC) P1484.1/D9 (2001). Draft Standard for Learning Technology<br />

Systems Architecture (LTSA), retrieved May, 30, <strong>2006</strong> from<br />

http://ltsc.ieee.org/wg1/files/IEEE_1484_01_D09_LTSA.pdf.<br />

IEEE Learning Technology Standards Committee (LTSC) WG12 (2003). LOM Meta Data Standard. URL:<br />

http://ltsc.ieee.org/wg12/.<br />

IMS (2004). IMS Content Packaging Information Model Version 1.1.3 Final Specification, retrieved May, 30, <strong>2006</strong> from<br />

http://www.imsglobal.org/content/packaging/index.cfm.<br />

IMSLD (2004), retrieved May, 11, <strong>2006</strong> from http://www.imsglobal.org/learningdesign/index.html.<br />

Isakowitz, T., Stohr, E. A. & Balasubramanian, P. (1995). RMM: A Methodology for Structured Hypermedia Design. Comm.<br />

ACM 38 (8).<br />

Klein, M. (2002). Courseware Engineering: Ein Vorgehensmodell zur Erstellung von wiederverwendbaren hypermedialen<br />

Kursen. (Courseware Engineering: A Procedural Model for the Development of Reusable Hypermedia Courses, in German)<br />

Ph.D. thesis, Universität Karlsruhe.<br />

Lieberherr, K. J. (1996). Adaptive Object-Oriented Software: The Demeter Method with Propagation Patterns. PWS<br />

Publishing Company.<br />

Macromedia (2004a). Authorware 7, retrieved May, 11, <strong>2006</strong> from http://www.macromedia.com/software/authorware/.<br />

112


Macromedia (2004b). Director MX 2004, retrieved May, 11, <strong>2006</strong> from http://www.macromedia.com/software/director/.<br />

Merrill, M. D., Li, Z. & Jones, M. K. (1990). Second Generation Instructional Design. Educational Technology 30 (2), 7-14.<br />

Merrill, M. D. (2000). First Principles of Instruction, retrieved May, 30, <strong>2006</strong> from<br />

http://projects.ict.usc.edu/itw/gel/MerrillFirstPrinciples02.pdf.<br />

Pawlowski, J. M. (2001). Das Essener-Lern-Modell (ELM): Ein Vorgehensmodell zur Entwicklung computerunterstützter<br />

Lernumgebungen. (The Essener-Learning-Model (ELM): A Procedural Model for the Development of Computer-Supported<br />

Learning Environments, in German) Ph.D. thesis, Universität Essen.<br />

SCORE (2004). SCORE Metadata, retrieved May, 11, <strong>2006</strong> from http://www.ipd.uka.de/SCORE/xsd/score_v1.xsd.<br />

SEI (Software Engineering Institute) (2004): Component-Based Software Development. retrieved May, 30, <strong>2006</strong> from<br />

http://www.sei.cmu.edu/str/descriptions/cbsd.html.<br />

Szyperski, C. (2003a). Component Technology: What, where, and how? In Proceedings of the 25th International Conference<br />

for Software Engineering, 684-693.<br />

Szyperski, C. (2003b). Component Software: Beyond Object-Oriented Programming. 2nd ed. ACM Press.<br />

Tennyson, R. D. & Rasch, M. (1995). Instructional System Development: The Fourth Generation. In: Automating<br />

Instructional Design: Computer-Based Development and Delivery Tools. Springer, 33-78.<br />

ToolBook (2004). ToolBook, retrieved May, 11, <strong>2006</strong> from http://www.toolbook.com/<br />

Uschold, M. & Gruninger, M. (1996). Ontologies, Principles, Methods and Applications. Knowledge Engineering Review, 11<br />

(2), 93-155.<br />

113


Bender, D. M., & Vredevoogd, J. D. (<strong>2006</strong>). Using Online Education Technologies to Support Studio Instruction.<br />

Educational Technology & Society, 9 (4), 114-122.<br />

Using Online Education Technologies to Support Studio Instruction<br />

Diane M. Bender<br />

College of Design, Arizona State University, P.O. Box 872105, Tempe, AZ 85287-2105, USA<br />

Tel: +1 480-965-8684<br />

diane.bender@asu.edu<br />

Jon D. Vredevoogd<br />

School of Planning, Design, and Construction, Michigan State University, 309 Human Ecology, East Lansing,<br />

MI 48824 USA<br />

Tel: +1 517-353-3054<br />

vredevoo@msu.edu<br />

ABSTRACT<br />

Technology is transforming the education and practice of architecture and design. The newest form of<br />

education is blended learning, which combines personal interaction from live class sessions with online<br />

education for greater learning flexibility (Abrams & Haefner, 2002). Reluctant to join the digital era are<br />

educators teaching studio courses (Bender & Good, 2003), who may be unaware of the possibilities and<br />

benefits of teaching with technology. The argument proposed in this study is that blended learning will<br />

enhance studio courses. Studios are unique learning environments embedded in an historical context. This<br />

article presents a process of infusing a traditional studio with online technologies. The result is a more<br />

streamlined course that enhances student learning, provides targeted instruction to individual students,<br />

serves a larger group of students than a traditional studio, and does not increase faculty workload.<br />

Keywords<br />

Online education, Studio, Faculty workload<br />

Introduction<br />

Technology is moving higher education from the traditional campus of “brick and mortar” to the electronic<br />

classroom of “wire and chip.” The technology with the greatest potential for impact on architectural education is<br />

the Internet. By the year <strong>2006</strong>, it is estimated there will be 900 million computers and other web-based<br />

appliances in use, providing universal access, multimedia resources, and interactive medium (Charp, 2000). The<br />

Internet has revolutionized the design process by allowing architects and designers to research new products,<br />

download specifications, access code information, transfer drawings, and even synchronously collaborate with<br />

colleagues from around the world. Technology is also having an impact on the process and culture of<br />

architectural education, as evident in previous research on digitally immersed classes and studios (Matthews &<br />

Weigand, 2001). The use of digital media is a logical addition to the traditional design studio.<br />

Technology has radically changed the way educators can exchange information with students. Higher education<br />

support systems have adapted to changes in technological innovation, but the studio has remained disturbingly<br />

constant. Pedagogical integration of digital media is critical to the success of future architecture and design<br />

education. With the ever-increasing need to communicate globally, distance is no longer a barrier to education<br />

(Matthews & Weigand, 2001). Based on several years of experimentation, the authors believe studio courses can<br />

be enhanced with online technologies. A popular format for teaching both in the classroom and online is blended<br />

learning. Blended learning involves both traditional face-to-face instruction, where both students and faculty are<br />

present at the same time and place, supplemented with asynchronous and/or synchronous communication via the<br />

Internet. In comparison to courses that are completely online, courses with a combination of online and face-toface<br />

interaction produce the same or better success rates, plus dropout rates are lower (Dziuban & Moskal,<br />

2004). The authors are not advocating technology as a substitute for the existing process, but as a means to<br />

enrich instruction in the design studio.<br />

There are many benefits of using online technologies, such as the accuracy and consistency of data. All students<br />

are guaranteed to receive the same presentation material and get the same view of the professor and material,<br />

unlike the front row advantage in the traditional classroom. Students appreciate the on-demand access of online<br />

classes, for it allows them flexibility in viewing course information at their convenience and as many times as<br />

they wish. This information may include course syllabus and outline, staff profiles and contact information,<br />

examples of projects, and other useful items typically provided to students in a traditional course. Electronic<br />

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the<br />

copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies<br />

are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by<br />

others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior<br />

specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.<br />

114


collaboration allows students to propose, experiment, refine, and resubmit ideas. The following study examines<br />

the impact of blended learning on the traditional studio, in particular the impact on student learning, accessibility<br />

of course material, and faculty workload. The goal is to increase the quality of interaction without losing the<br />

master-apprentice relationship established in the traditional studio.<br />

The Traditional Design Studio<br />

The concept of today’s architecture and design studios began with the French Royal Academy and continued<br />

with the methodologies of the Ecole des Beaux-Arts. It became traditional for schools in the United States to<br />

pattern their instruction in Beaux-Arts format, in which the studio is the central focus of the curriculum (see<br />

Figure 1). Studios (where drawing, debate, and analysis of design take place) are considered more of an active<br />

learning experience than a lecture-style classroom.<br />

Figure 1: A Typical Studio Environment<br />

The strength and assurance of the Beaux-Arts approach was influential on the creation of architecture and design<br />

programs in the United States in the early 20 th century (“Architectural Education”, 2000). Since that time, studio<br />

instruction has essentially remained unchanged. Students then and now attend a studio where instruction is<br />

delivered from master to apprentice within a small group setting. Faculty to student ratio can range from 12 to 24<br />

students per instructor, with an average of 17 to 1 (Ochsner, 2000; Design Futures Council, 2005). Studio classes<br />

may range from four to 12 hours per week, during which the instructor works with each student independently<br />

for short periods of time. It may common for a student to wait almost three hours for a few minutes of insight<br />

and direction. While waiting, a student may not have the opportunity to view and hear the critique addressing<br />

the work of classmates. The more students present in the studio, the less time can be spent with each student.<br />

Because of the long tradition of one-on-one instruction, faculties in the arts frequently generate significantly<br />

more weekly contact hours than faculty in other disciplines (Lawn, 1998).<br />

Perpetuated by past experience, design educators continue to teach with the Beaux-Arts method, even though<br />

there is little evidence to suggest that it is an effective means of instruction (Rapoport, 1983, 1984). Since<br />

architectural design is viewed as a subjective process, the defense of the studio instruction format has been very<br />

feeble and unsuccessful (Wooley, 1991). The current model of teaching and learning is not used because it is the<br />

“right way”, but because the method has worked for so long a time (Farrington, 1999).<br />

The traditional studio is but one method of teaching architecture and design. Other forms of instruction are used<br />

in combination with the studio, such as large lectures, small group sessions, and classes held in computer labs<br />

115


(see Figure 2). The incorporation of technologies commonly used in online education into the traditional studio<br />

can alter the interaction between the faculty and the student (or the master and apprentice). It is hypothesized that<br />

blending technology with traditional instruction will impact typical studio problems of high student to faculty<br />

ratio and high faculty workload.<br />

The Modified Design Studio<br />

Figure 2: Alternative Education Environments<br />

This case study focused on an introductory course in Structural Systems for integration with online teaching<br />

technologies. This course is offered at a large public state university and is offered to students in majors such as<br />

building construction, engineering, interior design, landscape architecture, and packaging. After years of<br />

teaching this course as a traditional design studio with multiple class sections, the authors sought to streamline<br />

the teaching process. This course was a logical choice in which to add online technologies, as the students were<br />

creating their projects with Computer-Aided Design (CAD) software. The existing six hours of studio times were<br />

divided into a one-hour weekly lecture by the faculty and four hours of computer lab time supervised by<br />

qualified teaching assistants. A course website was created to house course materials and the faculty members<br />

were available outside of lecture via email and web-based conferencing.<br />

The proposed model expands the educational opportunity to participate in the design critique to more students, in<br />

more places, and in more ways than ever before. Blended learning is being used in various educational<br />

environments outside architecture and design (Jonasson, 1997; Ravaglia, 2001). This form of mediated education<br />

brings many benefits to the studio. One limitation is the need for both parties to be physically present in the same<br />

location at the same time. The proposed model streamlines the interaction loop by incorporating technology in<br />

the teaching and delivery of feedback to the entire class of students at one time. The instructor can post course<br />

materials and students can access it from any place and at any time.<br />

Typical studio interaction involves one student requesting feedback and one faculty providing it directly to that<br />

student. Though individual feedback is beneficial for student learning, the lower faculty to student ratio typical<br />

in a studio is impractical. As educational institutions continue to increase enrollments by expanding their<br />

“territory” through online education, faculty must deal with larger courses, often as multiple sections of the same<br />

studio course. A typical class size for this course is 100 students. Traditionally, this would equate to five sections<br />

116


of 20 students with six contact hours each, for a total of 30 faculty contact hours per week. The new model pars<br />

this down to one hour of lecture by the faculty, plus time spent training teaching assistants, answering email,<br />

doing conference calls, and preparing online critiques, for a total of approximately five hours per week. Teaching<br />

assistants cover 20 hours of computer lab time each week.<br />

Figure 3: The Modified Design Studio Process<br />

The new model involves 12 steps (see Figure 3). As in a traditional studio, each student works toward<br />

completing one or more major projects for the semester. Each week, parts of the project are due and critiqued in<br />

lecture. In a study of traditional studio interactions, students often can choose when to work in isolation and<br />

when to be interact socially (James, 1996). In this course, computer lab time is available for students to work<br />

independently or in small groups; lecture time is for interaction with the whole class.<br />

Students draw their projects by hand and with CAD. Each student reports his weekly progress by scanning the<br />

drawing into an image file, or by capturing the CAD drawing off the computer screen by depressing the Print<br />

Screen button on his keyboard. The student then pastes the image onto a blank Microsoft PowerPoint © slide. He<br />

may resize and add text as necessary. Assignments are saved to Adobe’s Portable Document File (PDF) format.<br />

This allows the drawing file to be compressed, while retaining the layout and graphic presentation of the<br />

submission. Using available file transferring software, he submits a copy of the image to the course website. The<br />

goal is to use basic technology, which every student will have on his computer, such as the Print Screen key on a<br />

keyboard, or free software to watch MP3 movies and view PDF files. All assignments are time and date stamped<br />

when submitted electronically, keeping the students more accountable for submitting their work on time.<br />

The process then transfers to the instructor or teaching assistant, who assembles all the image files into a single<br />

collection representing student work for that week (see Figure 4). Student work is placed in alphabetical order by<br />

last name. This allows the faculty to compare current work with previous work to ascertain progress (or lack of).<br />

Confidentiality is maintained by removing all names from the slides.<br />

117


Figure 4: Example of Summary Image File<br />

Figure 5: Example of Creating Audio Critique<br />

The instructor reviews each student’s submission to generate a critique in the form of a 20- to 30-minute MP3<br />

audio file (see Figure 5). This audio critique can be recorded while speech recognition software is running in the<br />

background, such as Nuance’s Dragon Naturally Speaking. This provides a written text of the critique as an<br />

alternative form of feedback or to accommodate diverse student learning styles. Not every slide receives<br />

comments each week, as some comments are redundant or have little impact on the class as a whole. A student<br />

may receive comments on his individual project one week and not the following week. All comments are in one<br />

audio file. Students hear comments directed at both their project and those of their classmates. The audio file and<br />

summary image file are then uploaded to the course website, where they are archived and accessible throughout<br />

the remainder of the semester. This format maintains a permanent record of student work and participation.<br />

Finally, the student reviews both files prior to the next weekly lecture. He is able to note his individual progress<br />

in comparison to his peers, which can be used as a measure of progress toward course goals (James, 1996).<br />

118


Student Learning<br />

Integrating blended learning with the traditional studio can increase student learning. Technology provides<br />

several benefits to students. It can be used to present information in a variety of formats, accessible at all times,<br />

and leaving live class time for the intellectual communications that only people can provide. The one-hour<br />

weekly lecture focuses on interaction with and between the students, reviewing a specific project with the whole<br />

class or doing problem solving in informal groups. The instructor uses this time to gauge the progress of the<br />

whole class, which is an important component of other technology-enhanced courses (Gilbert, 1999). It was<br />

believed that combining face-to-face meetings and online education would better provide access to instructional<br />

materials, facilitate group work, enhance the quality of interaction, and provide a superior level of feedback to<br />

students.<br />

Positive factors of learning with technology such as accuracy and consistency are hard to surpass. All students<br />

viewing the lecture are guaranteed to receive the exact same lecture material, regardless of the instructional<br />

model implemented (Charp, 2000). All students get the same audio and video materials, unlike the front row<br />

advantage in conventional classroom situations (Romiszowski, 1988). Students who have difficulty grasping<br />

concepts from printed materials may be more at ease with learning experiences that engage all of their senses.<br />

Information technologies are predominantly visual rather than relying primarily on audio and text in the<br />

traditional classroom. Because of this, complex content can be conveyed more effectively due to the integration<br />

of multiple representations of material such a computer animations with audio and video. These alternative<br />

technologies can provide an additional degree of motivation, allowing students to express themselves in new and<br />

creative ways (Dias, 1999). In addition, students hear all critiques of all projects and benefit from feedback to<br />

their peers. In a traditional studio, individual faculty comments are rarely shared with the entire class.<br />

Faculty Workload<br />

Many instructors are integrating computers into their architecture and design curriculums (Budd, Vanka, &<br />

Runton, 1999; Andia, 2002). Like them, the authors believe a major advantage to adding online technologies to<br />

the curriculum is the efficiency of instruction. In the traditional studio, it may be common for a student to wait<br />

two or more hours for a few minutes of insight and direction. With blended learning, the student gets more<br />

succinct feedback in less time. In addition, he learns to communicate a creative design solution with the latest<br />

technologies used in today’s design offices. Becoming familiar with these technologies is an advantage to the<br />

student as it helps prepare him to progress in professional practice offers career opportunities in digital<br />

architecture fields (Proctor, 2003).<br />

This model of interaction provides a number of advantages to the studio instructor. First, there is an opportunity<br />

to see the range of responses before making comments. He can quickly ascertain the effectiveness of each<br />

assignment and make any necessary adjustments in the instructional format. Tracking student work from week to<br />

week can also be done easily. The history and process of design is clearly articulated through the layering of<br />

information. This allows the faculty to watch the project evolve from concept to final presentation (Norman,<br />

2001). Second, faculty fatigue is reduced because critique duplication and the amount of time necessary for<br />

giving and receiving project feedback substantially decrease. This is a common problem in large classes with<br />

multiple sections where critiques are limited by time and space (James, 1996). Third, the feedback is streamlined<br />

because the instructor spends less time repeating the same information. He can increase the turn-around time for<br />

feedback to larger groups of students. Several other advantages are provided in Table 1.<br />

Traditional Design Studio<br />

Table 1: A Comparison of the Two Instructional Models<br />

Modified Design Studio<br />

Assignments are introduced by the instructor and Assignments are introduced in the same manner but are<br />

submitted by the students during class time.<br />

submitted electronically 24-48 hours before class time.<br />

Class size is typically 15-20 students to one instructor Class size remains the same but the instructor can handle<br />

per section. The same instructor may oversee two or<br />

more sections.<br />

an increased number of sections with assistance.<br />

The individual critiques provided in class are seldom Students receive feedback via the audio critique and<br />

shared with other class members. Therefore, the same<br />

feedback may be repeated to several students within the<br />

same class period.<br />

group feedback during the weekly lecture period.<br />

The critic repeatedly corrects the same or similar student<br />

errors.<br />

The critic needs to address student errors only one time.<br />

119


The competitive nature of design classes limits the<br />

sharing and nurturing of ideas and causes many students<br />

to work independently.<br />

Students have to be physically present in studio when a<br />

critique is given. Unless recorded, it cannot be reviewed<br />

later time.<br />

Guest critics must travel to the class site during a<br />

specified class period to participate in critiques.<br />

Work done in the studio may turn into a collaborative<br />

effort between the student and the instructor, making it<br />

difficult to determine what part of the project the student<br />

has done. The instructor must be cautious about doing<br />

the assignment for the student.<br />

Limitations<br />

Design skills are strengthened by the student’s ability to<br />

build on the feedback of other students’ projects and<br />

feedback.<br />

Students can review critiques on demand and can<br />

“attend” the critique from remote locations.<br />

Guest critics can review the projects and provide an<br />

audio critique (independent or with others) at their<br />

convenience.<br />

The instructor does not guide any single student through<br />

the design process, but guides the class as a whole.<br />

It is apparent this form of instruction has limitations. With the array of necessary technology, the instructor must<br />

devote time to learning these tools. The authors of this study were self-taught. Likewise, time must be spent<br />

teaching the tools to the students so that they feel comfortable using them. This is particularly true for students<br />

and educators who are only knowledgeable of traditional studio instruction (Sagun, Demirkan, & Goktepe,<br />

2001). Data from the latest NEA faculty survey indicate the level of technical support is an important factor in<br />

determining whether a faculty member has positive or negative feelings toward various forms of distance<br />

education (NEA, 2000). There is an obvious need for faculty development offered by the institution.<br />

In the new model, the instructor is not able to spend much personal time with each individual student. Though<br />

student interaction is high in the computer lab sessions, the traditional studio setting may provide the students<br />

more opportunity to socialize and network than in this format. Depending on the submissions received, the<br />

faculty may or may not address individual student work each week. Some students may have difficulty gleaning<br />

insight from comments not made directly to them. In a national survey of online education faculty, the average<br />

number of students in their courses is 26 students (Schifter, 2000). When much larger classes are accommodated<br />

in this online format, a potential downside is that each student will receive less individualized attention from the<br />

faculty.<br />

This process has been successful for introductory and advanced CAD courses. However, this approach may not<br />

be appropriate for every course because online activities cannot replace hand-drawing and rendering exercises,<br />

or activities requiring textual artistic media. In addition, educators may prefer to balance the lecture and lab<br />

times in a different ratio to allow more personal time with students. While student expectations are consistent<br />

between this model and traditional studio, assignment submission procedures are different. All assignments are<br />

now submitted electronically, freeing the student from the physical restraints of time and place. There is also a<br />

shift from instructor time devoted to teaching, to time spent on course development and management. The<br />

development and administration of a new course using online technologies can take more time than traditional<br />

studio methods (Bender, Wood, & Vredevoogd, 2004). Once initial course development is complete, faculty<br />

workload decreases due to increased instructor experience (Visser, 2000).<br />

Future Research<br />

Several issues need further exploration. One issue is setting boundaries for implementing new technology into<br />

curriculum. Similar to other findings (Visser, 2000), the development time for this course was higher than a<br />

traditional course. The formal individual interaction time noted in this paper can also be misleading. Email is<br />

exchanged between faculty, teaching assistants and students for additional communication throughout the week,<br />

which has been found to contribute to a heavy instructor workload (Lehman, 1996). Unless carefully monitored,<br />

the instructor can find himself available to students 24 hours a day, seven days a week, and spending more time<br />

answering email, critiquing projects, and working with the teaching assistants than interacting in a traditional<br />

studio setting. These concerns lead to a need for research in the area of course management strategies.<br />

120


Issues of Acceptance<br />

With the implementation of online technologies, the traditional studio environment and the instructor’s workload<br />

have been altered with the proposed process. Though many design educators are reluctant to embrace teaching<br />

with technology, the authors believe the addition of online technologies enhances studio education. The<br />

proposed alternative approach to studio instruction may provide potential solutions to the unease and resistance<br />

of these other faculty. In order to change the culture of the existing studio environment, digital media must<br />

become a transparent tool for design inquiry. Instructors should give serious thought to the complex pedagogical<br />

issues underlying new methods of teaching and learning before implementing them in the curriculum.<br />

Though this form of blended learning provides many benefits to students and faculty, the personal component<br />

should not be eliminated from studio education. To rely solely on virtual instruction poses a serious risk, as “live<br />

instruction has to build upon the virtual part and vice versa” (Van Eijl & Pilot, 2003, p. 55). As rapid changes<br />

occur in architectural practice, student populations, and the design process, continued research is necessary in<br />

order to understand technology’s impact on architectural education.<br />

Conclusion<br />

This study demonstrates that blended learning can revolutionize instruction in the design studio. Student learning<br />

can be enhanced by having pertinent course material available online whenever students wish to access it.<br />

Project critiques can be delivered in both audio and text format. They can be reviewed at any time and as many<br />

times as needed. All students can view and hear projects and comments of the entire class, which is something<br />

that is often missing in studio.<br />

This model also exemplifies the ability to serve a larger body of students without increasing faculty workload.<br />

The student enrollments of architecture and design programs are increasing at a faster-than-average rate in<br />

comparison to other disciplines. It is anticipated this trend will continue through 2010, exacerbating the problem<br />

of increased faculty workload (ASID report, 2004). The use of technology in the studio allows faculty feedback<br />

to be less repetitive and more streamlined. More students can be accommodated in this model than the traditional<br />

studio. If blended learning can be used to positively impact student learning with the same amount of faculty<br />

effort, the integration of technology into the studio environment will be viewed in a more favorable light.<br />

References<br />

Abrams, G. & Haefner, J. (2002). Blending online and traditional instruction in the mathematics classroom. The<br />

Technology Source, retrieved 10 May <strong>2006</strong> from http://ts.mivu.org/default.asp?show=article&id=970.<br />

Andia, A. (2002). Reconstructing the effects of computers on practice and education during the past three<br />

decades. Journal of Architectural Education, 56 (2), 7-13.<br />

Architectural Education, (2000). A brief history. Association of Collegiate Schools of Architecture. Retrieved<br />

August 5, 2004 from https://www.acsa-arch.org/students/education.aspx.<br />

ASID Report. (2004). The interior design profession: Facts and figures. Washington, DC: American Society of<br />

Interior Designers.<br />

Bender, D. M. & Good, L. (2003). Attitudes of higher education interior design faculty toward distance<br />

education. Journal of Interior Design, 29 (1/2), 66-81.<br />

Bender, D. M., Wood, B. J. & Vredevoogd, J. D. (2004). Teaching time: Distance education versus classroom<br />

instruction. The American Journal of Distance Education, 18 (2), 103-114.<br />

Budd, J. Vanka, S. & Runton, A.(1999). The ID-online asynchronous learning network: A ‘virtual studio’ for<br />

interdisciplinary design collaboration. Digital Creativity, 10 (4), 205-214.<br />

Charp, S. (2000). Internet usage in education. T.H.E. Journal, 27 (10), 12, 14.<br />

Design Futures Council (DFC). (2005). 2005 Architecture & Design Schools Rankings Issue Overview, retrieved<br />

May 6, <strong>2006</strong> from http://www.di.net/article.php?article_id=374.<br />

121


Dias, L. B. (1999, November). Integrating technology: Some things you should know. Learning and Leading<br />

with Technology, International Society for Technology in Education, 27 (3), 10-13.<br />

Dziuban, C. & Moskal, P. (2001). Distributed learning program impact evaluation, retrieved 10 May <strong>2006</strong> from<br />

http://www.tltgroup.org/resources/F_Eval_Cases/UCF_DistribLearn.htm.<br />

Farrington, G.C. (1999 July/August). The new technologies and the future of residential undergraduate<br />

education. Educom Review, 34 (4). Retrieved May 28, 2005 from<br />

http://www.educause.edu/LibraryDetailPage/666?ID=ERM9949.<br />

Gilbert, J. (1999). But where is the teacher? Learning and Leading with Technology, 27 (2), 42-44.<br />

James, J. (1996). The construction of learning and teaching in a sculpture studio class. Studies in Art Education,<br />

37 (3), 145-159.<br />

Jonasson, J. (1997 June). The Internet: The educational medium of today. Educational Media International, 34<br />

(2), 88-93.<br />

Lawn, R. J. (1998). Integrating the arts and technology. Educational Technology, 38 (6), 56-59.<br />

Lehman, J. D. (1996). Computer networking in distance education (147-155). In Boschmann, E. (Ed.) The<br />

electronic classroom: A handbook for education in the electronic environment, London: Learned Information.<br />

Matthews, D. & Weigand, J. (2001). Collaborative design over the Internet. Journal of Interior Design, 27 (1),<br />

45-53.<br />

National Education Association (NEA). (2002). A survey of traditional and distance learning higher education<br />

members. Retrieved February 26, <strong>2006</strong> from http://www.nea.org/nr/nr000614.html.<br />

Norman, F. (2001 April). Towards a paperless studio. Proceedings from the ARCC Spring Research Conference<br />

at Virginia Tech, 85-91, retrieved 10 May <strong>2006</strong> from http://www.asu.edu/caed/arcc/articles/AAINDEX.PDF.<br />

Ochsner, J. K. (2000). Behind the mask: A psychoanalytic perspective on interaction in the design studio.<br />

Journal of Architectural Education, 53 (4), 194-206.<br />

Proctor, G. (2003). Digital Foundations: Building a base for digital futures. In Bermudez, J. & Klinger, K.<br />

(Eds.), Digital technology and architecture, Association for Computer Aided Design in Architecture, white<br />

paper.<br />

Rapoport, A. (1983 <strong>October</strong>). Studio questions. The Architects’ Journal, 55-57.<br />

Rapoport, A. (1984 <strong>October</strong>). Architectural education. Architectural Record, 100-105.<br />

Ravaglia, R. (2001 August/September). E-education: The case for blended e-learning. Education West, 12-13.<br />

Romiszowski, A. J. (1988). The Selection and Use of Instructional Media (2 nd Ed.). New York: Nichols.<br />

Sagun, A., Demirkan, H. & Goktepe, M. (2001). A framework for the design studio in web-based education.<br />

Journal of Art and Design Education, 20 (3), 332-342.<br />

Schifter,C. C. (2000). Compensation models in distance education. Online Journal of Distance Learning<br />

Administration, 3 (1), retrieved 10 May <strong>2006</strong> from http://www.westga.edu/~distance/schifter31.html.<br />

Van Eijl, P. & Pilot, A. (2003). Using a virtual learning environment in collaborative learning: Criteria for<br />

success. Educational Technology, 43 (2), 55.<br />

Visser, J. A. (2000). Faculty work in developing and teaching web-based distance courses: A case study of time<br />

and effort. The American Journal of Distance Education, 14 (3), 21-32.<br />

Wooley, T. (1991). Why Studio? The Architects’ Journal, 20, 46-49.<br />

122


Montazemi, A. R. (<strong>2006</strong>). The Effect of Video Presentation in a CBT Environment. Educational Technology & Society, 9 (4),<br />

123-138.<br />

The Effect of Video Presentation in a CBT Environment<br />

Ali Reza Montazemi<br />

DeGroote School of Business, McMaster University, 1280 Main St. West, Hamilton, ON L8S 4M4, Canada<br />

Tel: +1 905 529 7070 Ext. 27434<br />

Fax: +1 905 521 8995<br />

montazem@mcmaster.ca<br />

http://www.business.mcmaster.ca/msis/profs/montaz<br />

ABSTRACT<br />

Multimedia is increasingly used to enhance human-computer interaction in a variety of formats (text,<br />

graphics, audio, animation, and video). The challenge is how to use the most effective presentation format<br />

that would result in performance improvement. This paper addresses the added value of video<br />

presentations in a technology-mediated learning environment for a basic Management Information System<br />

course at MBA level. We postulate two hypotheses that take into account students’ mastery learning and<br />

satisfaction with regard to the usefulness of video presentation, moderated by students’ self-interest to learn<br />

the subject matter (i.e., intrinsic motivation). Our analysis, on the basis of empirical research, shows that<br />

the subjects find a learning environment supported with video presentations is more useful. Their intrinsic<br />

motivation to learn the subject matter had a significantly positive effect on their satisfaction with the<br />

availability of video presentations. Video presentations had no significant effect on their performance on<br />

exams, however.<br />

Keywords<br />

Video presentation, Multimedia, CBT<br />

Introduction<br />

The most significant applications of multimedia are in (1) education and training, (2) communications,<br />

coordination, and collaboration, and (3) entertainment. With the growth of the new market for multimedia, end<br />

users, application developers, and content providers face the challenge of how to effectively manage, organize,<br />

and access the vast amount of information they accumulate. The focus of this paper is on the effect of video<br />

presentation for education and training in the context of technology-enabled learning media. There is an<br />

increasing expectation that information technology will enhance the learning process (Benbunan-Fich, 2002).<br />

This expectation is based on the notion that information technology, by supporting interactive instruction, will<br />

encourage students to be more responsible for their own learning. In this endeavor, the design of information<br />

technology should be based on well-grounded theories of learning processes (Alavi and Leidner, 2001; Hannafin<br />

and Rieber, 1989; Leidner and Jarvenpaa, 1995). For example, Hannafin and Rieber (1989) caution, “The<br />

emergence of computer-driven ‘hybrid’ technologies … has spawned unprecedented interest, yet advances in<br />

technological capability alone no more improve instruction than sharpened pencils improve prose.” More than a<br />

decade later, a research essay by Alavi and Leidner (2001) calls for greater depth and breadth of investigation<br />

into technology-mediated learning. It seems that in spite of the large number of reported research findings, we<br />

know little about the effective means of applying and managing information technology in support of the<br />

learning process. This is mainly due to the intricacy of the task at hand, for the dynamic nature of the interaction<br />

between the environmental factors (i.e., instructional strategy and information technology) that affect the<br />

learning process is complex (Alavi and Leidner, 2001; Carroll, 1997; Leidner and Jarvenpaa, 1995).<br />

For example, let us consider the usefulness of dynamic visuals such as video presentations when used in support<br />

of learning. Video presentation is used routinely in support of classroom lectures to enhance the learning<br />

process. Inclusion of video in support of learning is expensive, and this means that we must make sure that it<br />

benefits the learner. This paper presents the findings from an empirical investigation into the effectiveness of<br />

video in a computer-based tutoring environment.<br />

Computer-based tutoring (CBT) systems have been used in diverse learning environments to enable the learner<br />

to self-pace in acquiring the pertinent materials. In such systems, feedback is a critical part of effective learning,<br />

and it is expected that active involvement will lead to more effective learning than will passive involvement.<br />

CBT systems are generally based on the stimulus-response-feedback views of learning that are associated with<br />

the objectivist model of learning (Leidner and Jarvenpaa, 1995; Montazemi and Wang, 1995-a; Retalis and<br />

Papasalours, 2005; Stone, 2001). Thus, CBT can be viewed as enhancing the cognitive information processing<br />

of learners by tailoring the learning process to individual needs (Leidner and Jarvenpaa, 1995; Montazemi and<br />

Wang, 1995-b). The learner is guided by the CBT system so that he or she reaches the predefined learning<br />

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the<br />

copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies<br />

are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by<br />

others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior<br />

specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.<br />

123


materials built into the system. Therefore, CBT systems can be developed for learning environments with a<br />

well-defined structure, pre-defined learning objectives and possible solutions to the learner’s enquiries.<br />

Multimedia technology can enrich the learning experience by making it possible to present the learning materials<br />

in different formats. The question raised in this paper is whether inclusion of video presentations in a CBT<br />

system is useful to the learner. To this end, the following two hypotheses are postulated and empirically tested:<br />

HYPOTHESIS 1: Inclusion of video in a CBT system has no effect on the learner's perception regarding the<br />

usefulness of the system.<br />

HYPOTHESIS 2: Inclusion of video in a CBT system has no effect on the learner's mastery of the material to be<br />

learned.<br />

The remainder of paper is structured as follows: Section Two provides a review of research findings related to<br />

the usefulness of dynamic visuals to the learning process. In Section Three, the basic theories on multimedia<br />

learning and human perception are used to postulate above hypotheses in regard to the usefulness and<br />

effectiveness of dynamic visuals in support of the learning process. A CBT system called E-Tutor is used to test<br />

these hypotheses. Section Four outlines the methodology adopted in this investigation. Section Five presents<br />

the results. A discussion and an overview of the implication for practice and future research close the paper.<br />

On the Usefulness of Dynamic Visuals on Learning<br />

Effective presentation of instructional materials, whether it be in the form of text, picture, charts, graphs,<br />

diagrams, animation, videos, etc. is an important facet of the learning process. As a result, multimedia<br />

educational systems are rapidly growing in popularity because they enable us to provide a variety of presentation<br />

modes in support of the learning process (Anderson et al., 2000; Heller et al., 2001; Pilkington and Grierson,<br />

1996). In particular, interactive multimedia systems with dynamic visual presentations are becoming popular<br />

because they can provide an enriched learning environment by facilitating the acquisition of pertinent materials.<br />

Dynamic visuals that have been embedded in CBT for improved learning can be classified as movies and<br />

simulations. We can distinguish movies and simulations in the following way: animations, videos, time-lapse<br />

photography and other motion pictures that are the same each time they are viewed are classified as movies; and<br />

graphics, tables, or other output generated by running a computer program under student control are classified as<br />

simulations (Pane, 1994). In the latter, the visual results are subject to any changes that the student makes to the<br />

simulation program, and thus may be different each time they are viewed.<br />

A number of investigators have examined the effect of dynamic visuals on learning (Pane et al., 1996; Park and<br />

Hopkins, 1993; Rieber, 1990; Velayo, 2000). Park and Hopkins (1993) identified five important instructional<br />

roles of dynamic visuals:<br />

1. As an attention guide, the dynamic visual can serve to guide and direct the subject's attention.<br />

2. As an aid for illustration, dynamic visuals can be used as an effective aid to represent the structural and<br />

functional relations among components in a domain of knowledge.<br />

3. As a representation of domain knowledge, movement and action can be used to effectively represent certain<br />

domain knowledge.<br />

4. As a device model for forming a mental image, dynamic visuals can be used to represent system structures<br />

and functions that are not directly observable (e.g., blood flowing through the heart).<br />

5. As a visual analogy or reasoning anchor for understanding abstract and symbolic concepts or processes,<br />

dynamic visuals can make abstract and symbolic concepts (e.g., velocity) become more concrete and<br />

directly observable.<br />

The question arises whether dynamic visual has any effect on students' learning. To answer this, Park and<br />

Hopkins (1993) produced a research summary of 25 studies and concluded that:<br />

The research findings do not consistently support the superior effect of dynamic visual displays.<br />

The conflicting findings seem to be related to the different theoretical rationales and<br />

methodological approaches used in various studies (Park and Hopkins, 1993, p. 427).<br />

Traditionally, designers of instructions have had two categories of outcomes in mind: those directed toward<br />

cognitive goals, and those related to the attitude of the learner. Student achievement related to cognitive goals<br />

has been the paramount objective of most instructional activities. However, it may also be important to recognize<br />

the need for establishing attitudinal goals and for planning activities that are designed to facilitate the learning<br />

process. Thus, one of the major goals of instruction involving media should be the development of positive<br />

124


attitudes (Venkatesh, 1999; Simonson, 1985; Simonson and Maushak, 1995). Although the strength of the<br />

relationship between attitudes and achievement is unclear (Zimbardo and Leippe, 1991), it seems logical that<br />

students are more likely to remember information, seek new ideas, and continue studying when they react<br />

favorably to an instructional situation or when they like a certain content area. Nonetheless, despite its<br />

importance in the learning process, we know little about assessment of student perception on the usefulness of<br />

dynamic visual media and their effect on learning. This vital research question is addressed in this paper.<br />

The Objective of this Investigation<br />

The basic question raised is "How do dynamic visuals in the form of videos embedded in a CBT system enhance<br />

learning?" As mentioned above, previous investigations have contradictory findings concerning improvement in<br />

students' learning outcomes (i.e., improved exam scores) when dynamic visuals were added to the CBT systems.<br />

Missing from these studies is some consideration of the internal psychological processes through which learning<br />

occurs (Alavi and Leidner, 2001). The contention of Alavi and Leidner in regard to technology-mediated<br />

learning (TML) is that:<br />

To be useful, TML research questions need to be formulated in terms of the way in which<br />

technology features can engage psychological processes of learning that will in turn result in the<br />

desired learning outcomes (Alavi and Leidner, 2001, p. 4).<br />

Dynamic visuals can be embedded in CBT systems in the form of multimedia learning instructions. Multimedia<br />

learning occurs when students use information presented in two or more formats – such as text, figures, and<br />

videos. Figure 1 summarizes a dual-coding theory of multimedia learning (Mayer and Sims, 1994). It offers a<br />

three-process account of how visually and verbally presented material might be integrated in the learner’s<br />

working memory. On the top left portion of the figure, a verbal explanation, such as an oral narration, is<br />

presented to the learner. Within the working memory the learner constructs a mental representation of the<br />

system described in the verbal explanation. The cognitive process of going from an external to an internal<br />

representation of the verbal material is called building a verbal representational connection (or verbal encoding)<br />

(Mayer and Sims, 1994). On the bottom left portion of the figure, a visual explanation is presented to the<br />

learner, such as video. Within the working memory the learner constructs a mental representation of the visually<br />

presented system. The cognitive process of going from an external to an internal representation of visual<br />

information is called building a visual representational connection (or visual encoding) and is indicated by the<br />

second arrow. The third arrow denotes the construction of referential connections between the two mental<br />

representations, that is, the mapping of structural relations between the two representations of the system. Longterm<br />

memory enables the learner to interpret and assimilate the content of working memory to interact with the<br />

decision environment. It is expected that the added value of multimedia enhance the decision performance of the<br />

learner. For example, in understanding an explanation of how intranet and extranet work, the learner can build<br />

referential connections between visual and verbal representation of essential parts, actions, relations and<br />

principals in the communication system.<br />

Figure 1: A dual coding model of multimedia learning (Mayer and Sims, (1994)<br />

125


This theory of multimedia learning has been the basis of previous investigations on the effectiveness of dynamic<br />

visuals in support of the learners’ improved learning performance. However, as indicated earlier, missing from<br />

these investigations is internal psychological processes of the learner. Students must be motivated to use<br />

strategies to regulate their cognition and effort (Pintrich, 1988 & 1989). Research findings suggest that students<br />

with a motivational orientation involving goals of mastery learning, as well as a belief that the task is interesting<br />

and important, will engage in more metacognitive activity, more cognitive strategy use and more effective effort<br />

management (Ames and Archer, 1988; Meece et al., 1988; Nolen, 1988). There are two main classes of<br />

motivation: extrinsic and intrinsic (Davis et al., 1992). Extrinsic motivation relates to the drive to perform a<br />

behavior to achieve specific goals/rewards, while intrinsic motivation relates to perception of pleasure and<br />

satisfaction from performing the behavior.<br />

In this research, our measurement for CBT system's learning outcome is the learner’s performance and affective<br />

reactions to the usefulness of videos. We use exam scores as a measure of the student’s extrinsic motivation to<br />

achieve mastery learning of the pertinent learning materials. A student’s affective reaction to the usefulness of<br />

videos is based on the Technology Acceptance Model (TAM) (Davis, 1989; Davis et al., 1989, 1992; Taylor and<br />

Todd, 1995). In TAM, extrinsic motivation and the associated instrumentality are captured by the perceived<br />

usefulness construct (Davis et al., 1992). In this research, affect of dynamic visuals on extrinsic motivation of<br />

the learner and subsequent learning outcome is of interest. To this end, based on previous research findings, the<br />

following two null-hypotheses are postulated:<br />

HYPOTHESIS 1: Inclusion of video in a CBT system has no effect on the learner's perception regarding the<br />

usefulness of the system.<br />

HYPOTHESIS 2: Inclusion of video in a CBT system has no effect on the learner's mastery of the material to be<br />

learned.<br />

An effective assessment of these hypotheses requires us to take into account the effect of the learner’s intrinsic<br />

motivation and self-regulated learning behavior, in form of covariates, which are important aspects of learning<br />

and academic performance (Corno and Mandinach, 1983; Corno and Rohrkemper, 1985). The theoretical<br />

framework adopted in this research for conceptualizing student intrinsic motivation is an adaptation of a general<br />

expectancy-value model of motivation (Pintrich, 1988, 1989; Pintrich and DeGroot, 1990). The model<br />

proposes that there are three motivational components that can be linked to the components of self-regulated<br />

learning: (a) an expectancy component, which includes the student’s belief in his or her ability to perform a task<br />

(i.e., self-efficacy), (b) a value component, which includes the student’s goals and beliefs about the importance<br />

and interest of the task (i.e., intrinsic value), and (c) an affective component, which includes the student’s<br />

emotional reactions to the task (i.e., test anxiety).<br />

Self-regulated learning adopted in this research as covariate to assess the stated hypotheses includes the student’s<br />

metacognitive strategies for planning, monitoring, and modifying cognition (Corno, 1986; Weinstein and Mayer,<br />

1986; Zimmerman, 1988). Different cognitive strategies have been found to foster active cognitive engagement<br />

in learning and to result in higher levels of achievement (Weinstein and Mayer, 1986). Self-regulation, on the<br />

other hand, refers to the student’s ability to manage and control the handling classroom academic tasks. For<br />

example, capable students who persist at a difficult task or block out distractions (i.e., noisy classmates) maintain<br />

their cognitive engagement in the task, and so are able to perform better (Corno, 1986; Corno and Rohrkemper,<br />

1985).<br />

Methodology<br />

A CBT system called E-Tutor, developed in support of a Management Information Systems (MIS) textbook was<br />

adopted to test the stated hypotheses (see O’Brien, 2001). The structure of E-Tutor was similar to the model<br />

proposed by Retalis and Papasalours (2005). E-Tutor enables each student to test his or her knowledge of the<br />

lessons to be learned. Textbook materials are embedded in the E-tutor in the form of hypermedia that presents<br />

text, pictures, figures and videos. We developed two versions of E-Tutor referred here as “without-video” and<br />

“with-video”: both versions had the basic textbook materials (i.e., text, figures, tables and pictures). However,<br />

one version had 80 videos in form of AVI files, called here E-Tutor (with video), and the other without videos,<br />

called here E-Tutor (without video). The videos that were embedded in the E-Tutor (with video), provided<br />

further definition or examples to support various parts of the text of the lessons being studied. These videos are<br />

expected to add value to the instructor’s explanation of the materials. Thus, McGraw Hill has included them in<br />

the instructor's CD ROM that accompanies their MIS textbooks. The question posed through our hypotheses is<br />

126


whether such learning materials in the form of videos are useful to the learning process when embedded in a<br />

CBT system.<br />

Experimental Design<br />

We used a 2x2 factorial design with one between-subjects factor and two repeated trials to assess the stated<br />

hypotheses. The between-subject factor was the presence or absence of the AVI videos embedded in a CBT<br />

system. Two dependent variables used were the usefulness of the video presentations in support of the learning<br />

materials and students’ test score gained from two exams. Students’ intrinsic motivation to learn the subject<br />

matter and self-regulated learning behavior, assessed through two structured questionnaires, were used as<br />

covariates.<br />

Subjects<br />

Fifty-one students registered in a basic Management Information Systems course at the MBA level were asked to<br />

participate in this investigation by completing the following steps for 5% extra credit towards their final grade.<br />

Student participation was optional and they could drop out at will. Forty-eight students completed the<br />

experiments. None of the subjects were disabled.<br />

Procedures<br />

There were four exams, three weeks apart, for the students to take. Each exam, which consisted of 50 multiple<br />

choice questions with medium-level of difficulty, was to cover 3-4 chapters of the textbook materials. The<br />

results of the first two exams were used to assess random assignment of the students to two groups (See<br />

Appendix A). Assessment of the stated hypotheses was based on exams 3 and 4.<br />

A 44-item questionnaire, adopted from Pintrich and DeGroot (1990), to assess the learner’s “motivation” and<br />

“self-regulated learning” (Appendix B) was administered when the class had covered half the textbook and taken<br />

the first two exams. By covering about half of the textbook materials and completing two exams, students were<br />

expected to have a better understanding of their interest in the subject matter.<br />

Next, students were randomly placed into two groups (Table 1). Each student in one group received a CD that<br />

contained E-Tutor (with video), and each subject in the other group received a CD that contained E-Tutor<br />

(without video). Students participating in this investigation were required to use E-Tutor to prepare for the two<br />

scheduled exams 3 and 4. This repeated trials made it possible to assess the effects of videos embedded in the E-<br />

Tutor within and between groups. A hands-on demonstration of E-Tutor use was done to make sure that all<br />

students were familiar with the way the system functioned.<br />

Table 1: Assignment of the subjects to the two test groups<br />

Treatment Trial 1 Trial 2<br />

Subject with access to E-Tutor (with Video) 25 23<br />

Subjects with access to E-Tutor (without video) 23 25<br />

Trial 1: Subjects were required to complete the pertinent chapters in E-Tutor in order to prepare for the<br />

scheduled third exam. In addition, subjects who had access to E-Tutor (with video) were required to view all the<br />

pertinent videos. E-Tutor kept track of each student’s progress through the lessons and saved it in a “working<br />

file”. To make sure that all the subjects completed the assigned tasks, they were required to upload their<br />

working files to a site. This upload included captured data about the date that the assigned chapters were<br />

completed, as well as the video files—for students with E-Tutor (with video)—that were viewed. Two weeks<br />

after receiving the E-Tutor CDs, students took a scheduled exam that consisted of 50 multiple-choice questions<br />

covering the assigned chapters of the textbook (Appendix C). Prior to viewing the questions, they completed a<br />

questionnaire to assess the usefulness of E-Tutor in support of related learning materials for the exam (Appendix<br />

D).<br />

Trial 2: The procedure followed in this trial was similar to trial 1, except that subjects who used E-Tutor (with<br />

video) in trial 1 exchanged it for E-Tutor (without video). Those with E-Tutor (without video) exchanged it with<br />

127


E-Tutor (with video). Two weeks after exchanging the E-Tutor CDs, students took the fourth exam. Prior to<br />

viewing the questions on this exam, as in trial 1, subjects completed a questionnaire designed to assess their<br />

perceived usefulness of videos.<br />

Results<br />

We used MANOVA to assess within and between subject difference in regard to the E-Tutor’s usefulness with<br />

and without videos in support of learning and improved performance. Students’ “motivation” and “selfregulated<br />

learning” were included in the analysis as covariates. Levene’s test of equality of error variance of the<br />

dependent variables showed insignificant inequality across the groups (for usefulness F=0.49 (p=0.69) and for<br />

the tests 3 & 4 F=0.93 (p=0.43)). In addition, Box’s test of equality of covariance matrices showed insignificant<br />

inequality across groups (F=0.78 (p=0.63)). Hypothesis 1 states, “Inclusion of video in a CBT system has no<br />

effect on the learner's perception regarding the usefulness of the system”. The average score for the E-Tutor<br />

usefulness (Table 2) in the Trial 1 was: with video = 30.24 and without video = 27.64, and for the Trial 2: with<br />

video = 30.97 and without video = 28.39. The results of MANOVA test demonstrated that there was a<br />

significant difference (F=11.43), p


Our hypothesis 2 states; “Inclusion of videos in a CBT system has no effect on the learner's mastery of the<br />

material to be learned.” The results of MANOVA test (Table 3) showed that there was no significant difference<br />

between the scores on exams 3 and 4 with and without video (F = 0.56, p = 0.64). This indicates that although<br />

availability of video was perceived to be useful in students’ learning processes, it had no effect on their mastery<br />

of the material to be learned based on their exam scores. Thus, hypothesis 2 cannot be rejected.<br />

Table 4: Pairwise Comparisons of Usefulness of E-Tutor with and without Video<br />

Dependent Variable: Usefulness<br />

Mean Difference Sig. (p)<br />

(I) FACTOR vs. (J) FACTOR<br />

(I-J)<br />

Group with video in trial 1 vs. group without video in trial 1 2.60 0.00<br />

Group with video in trial 2 vs. group without video in trial 2 test 2.52 0.00<br />

Group with video in trial 1 vs. group with video in trial 2 -0.61 0.35<br />

Group without video in trial 1 vs. group without video in trial 2 -0.69 0.29<br />

Discussion<br />

Inspired by the advent of new information and communication technologies, use of technology-mediated<br />

learning environments is on the rise. The number of online courses offered through academic and professional<br />

institutions is evidence of this. Although research related to technology-based learning has been with us for<br />

more than 30 years, we are still at an early stage of knowing how to effectively apply technology in support of<br />

learning. In this endeavor, textual material is supplemented by still images, animation, video sequences, and/or<br />

sound sequences. The objective is to produce an enhanced learning environment in which students will be able<br />

to learn and understand more effectively. This research shows that students find a learning environment with<br />

video presentation more useful. Furthermore, it shows that students’ intrinsic motivation to learn the subject<br />

matter had a significant positive effect on their satisfaction with the availability of video presentation. Based on<br />

this, we can conclude that students with higher interest in the subject matter would have found the added<br />

information gleaned from video presentation more satisfying. However, added functionality of video<br />

presentation had no significant effect on performance in exams. That is, the addition of the video presentation to<br />

the text and pictures did not produce significant gains in students’ ability to answer questions on it. Of course,<br />

these findings are limited by the characteristics of our environment. E-Tutor was designed to support text-based<br />

explicit knowledge to improve student mastery learning of basic MIS materials. However, the effect of video<br />

presentation on learning might be different when used in support of tacit knowledge (e.g., analysis of complex<br />

case studies) (Lim and Benbasat, 2000). Tacit knowledge relate to experiential learning, whereas explicit<br />

knowledge deals with objective and technical knowledge. Thus, future research should assess the effectiveness<br />

of video in support of learners in need of acquiring tacit knowledge.<br />

Future research should find it interesting to assess the long-term effect of video on students learning and student<br />

motivation to pursue the subject matter further. Learners’ cognitive style may also affect effectiveness of video<br />

presentation (Montazemi and Wang, 1989). In this endeavor, we need to develop methodologies to distinguish<br />

students as visual-spatial learners or auditory-sequential learners: Visual-spatial learner is believed to think in<br />

pictures and have a different brain organization than auditory-sequential learner (Silverman, 2002). It is possible<br />

that visual-spatial learns benefit more from video presentation. We should also search for different means of<br />

using dynamic visuals in support of improved learning performance. Learning performance, as depicted in<br />

Figure 1, is contingent on internal (long-term memory) and external (e.g., video) mental representation of the<br />

learner. Dynamic visual based on animation is a possible medium to satisfy the individual learning requirements<br />

of the learner. We can exploit the rapidly evolving graphics capabilities of information systems to provide<br />

learners with engaging three-dimensional worlds (Badler et al., 2002). For example, to learn procedural tasks, a<br />

three-dimensional learning environment could enable students to perform the task directly in the environment.<br />

Hence, rather than memorizing an abstract procedure, students learn by interacting with rich three-dimensional<br />

models representing the subject matter. Lifelike pedagogical agents can play a key role in a central problem<br />

posed by task-oriented three-dimensional learning environment; detecting and correcting misconception. While<br />

the pertinent technologies are new, they are beginning to mature, and we have already begun to see the<br />

emergence of techniques for real-time behavior sequencing in response to rapidly changing learning contexts<br />

(Badler et al., 2002; Funge, 2000; Light and Maybury, 2002).<br />

Information technology can play a significant role in support of learning if the embedded functionality of the<br />

system is perceived to be useful by the learner. We believe, as others do (Alavi and Leidner, 2001; Carroll,<br />

1997; Ohlsson, 1991; Tong and Angelides, 2000), that system “hacking” is not an effective means of improving<br />

the state of the art in technology-mediated learning (TML). Instead, well-grounded learning theories need to be<br />

129


empirically tested in TML environments. Only then will system designers be provided with the much-needed<br />

theoretical foundation for implementing more perfect TML systems.<br />

Acknowledgement<br />

The Author would like to thank the three anonymous reviewers for their comments on the earlier version of this<br />

manuscript. This research has been supported in part by grants from Indiana University Ameritech Fellow of<br />

Information Technology Program, and Natural Sciences and Engineering Research Council of Canada.<br />

References<br />

Alavi, M. & Leidner, D. E., (2001). Research Commentary: Technology-Mediated Learning-A Call for Greater<br />

Depth and Breath of Research. Information Systems Research, 12 (1), 1-10.<br />

Ames, C. & Archer, J., (1988). Achievement Goals in the Classroom: Student Learning Strategies and<br />

Motivation Processes. Journal of Educational Psychology, 80 (4), 260-267.<br />

Anderson, A. H., Smallwood, L., MacDonald, R., Mullin, J., Annemarie, F. & O’Malley, C., (2000). Video Data<br />

and Video Links in a Mediated Communication: What Do Users Value? International Journal of Human-<br />

Computer Studies. 52 (1), 165-187.<br />

Badler, N. I., Erignac, C. A. & Liu, Y., (2002. Virtual Humans for Validating Maintenance Procedures.<br />

Communications of the ACM. 45 (7), 57-63.<br />

Benbunam-Fich, A, (2002). Improving Education and Training with IT,” Communications of the ACM. 45 (5),<br />

94-99.<br />

Carroll, J. M., (1997). Human-Computer Interaction: Psychology as a Science of Design. International Journal<br />

of Human-Computer Studies. 46 (4), 501-522.<br />

Corno, L., (1986). The Metacognitive Control Components of Self-Regulated Learning. Contemporary<br />

Educational Psychology, 11 (4), 333-46.<br />

Corno, L. & Mandinach, E., (1983). The Role of Cognitive Engagement in Classroom Learning and Motivation.<br />

Educational Psychologist, 18 (4), 88-100.<br />

Corno, L. & Rohrkemper, M., (1985). The Intrinsic Motivation to Learn in Classrooms,” In Ames, C. & Ames,<br />

R. (Eds.), Research in Motion, Vol. 2. The Classroom Milieu, Academic Press, New York, NY, 53-90.<br />

Davis, F. D., (1989). Perceived Usefulness, Perceived Ease of Use and User Acceptance of Information<br />

Technology. MIS Quarterly, 13 (3), 319-39.<br />

Davis, F. D., Bagozzi, R. P. & Warshaw, P. R., (1992). Extrinsic and Intrinsic Motivation to Use Computers in<br />

the workplace. Journal of Applied Social Psychology, 22 (4), 1111-32.<br />

Davis, F. D., Bagozzi, R. P. & Warshaw, P. R., (1989). User Acceptance of Computer Technology: A<br />

Comparison of Two Theoretical Models. Management Science, 35 (8), 982-1002.<br />

Funge, J., (2000). Cognitive Modeling for Games and Animation. The Communications of the ACM. 43 (7), 41-<br />

48.<br />

Hannafin, M. J. & Rieber, L. P., (1989). Psychological foundations of instructional design for emerging<br />

computer-based instructional technologies. Part II. Educational Technology Research and Development, 7, 102-<br />

14.<br />

Heller, R. C., Martin, C. D., Haneef, N. & Gievska-Krliu, S., (2001). Using a Theoretical Multimedia Taxonomy<br />

Framework. ACM Journal of Educational Resources in Computing, 1 (1), 1-22.<br />

130


Kerlinger, F. N., (1973). Foundations of Behavioral Research. Holt, Rinehart, and Winston Inc., Chicago, IL.<br />

Leidner, D. E. & Jarvenpaa, S. L., (1995). The Use of Information Technology to Enhance Management School<br />

Education: A Theoretical View. MIS Quarterly, 19 (3), 265-91.<br />

Light, M. & Maybury, M. T., (2002). Personalized Multimedia Information Access. Communications of the<br />

ACM, 45 (5), 54-59.<br />

Lim, K. H. & Benbasat, I, (2000). The Effect of Multimedia on Perceived Equivocality and Perceived<br />

Usefulness of Information Systems. MIS Quarterly, 24 (3), 449-471.<br />

Mayer, R. E. & Sims, V. K., (1994). For Whom Is a Picture Worth a Thousand Word? Extensions of a Dual-<br />

Coding Theory of Multimedia Learning. Journal of Educational Psychology, 86 (3), 389-401.<br />

Meece, J., Blumenfeld, P. & Hoyle, R., (1988). Students’ Goal Orientations and Cognitive Engagement in<br />

Classroom Activities,” Journal of Educational Psychology, 80 (4), 514-23.<br />

Montazemi, A. R. & Wang, F., (1995-a). CBT in Support of Mastery Learning. Journal of Educational Computing<br />

Research, 13 (2), 185-205.<br />

Montazemi, A.R. & Wang, F., (1995-b). On the effectiveness of Neural Network for Adaptive External Pacing.<br />

Journal of Artificial Intelligence in Education, 6 (4) 379-404.<br />

Montazemi, A. R. & Wang, S., (1989). The Effects of Modes of Presentation on Decision Making: A Review and<br />

Meta-Analysis. Journal of Management Information Systems, 5 (3), 111-127.<br />

Nolen, S. (1988). Reasons for Studying: Motivational Orientations and Study Strategies,” Cognition and<br />

Instruction, 5 (4), 269-287.<br />

O’Brien, J. A., (2001). Management Information Systems; Managing Information Technology in the E-Business<br />

Enterprise. New York, NY, McGraw-Hill.<br />

Ohlsson, S., (1991). System Hacking meets learning Theory: Reflection on the Goals and Standards of Research<br />

in Artificial Intelligence and Education. Journal of Artificial Intelligence in Education, 2 (3), 5-18.<br />

Pane, J. F., (1994). Assessment of the ACSE Science Learning Environment and the Impact of Movies and<br />

Simulations,” Carnegie Mellon University, CMU-CS-94-162.<br />

Pane, J. F., Corbett, T. A., & John, B. E., (1996). Assessing Dynamics in Computer-Based Instruction. In<br />

Proceedings of the Conference on Human Factors and Computing Systems, April 13-18, Vancouver, Canada,<br />

197-204.<br />

Park, O. & Hopkins, R., (1993). Instructional Conditions for Using Dynamic Visual Displays: A Review.<br />

Instructional Science, 21 (6), 427-499.<br />

Pintrich, P. R., (1988). A Process-Oriented View of Student Motivation and Cognition. In Stark, J. S. &Mets, L.<br />

(Eds.), Improving Teaching and Learning Through Research. New Directions for Institutional Research, San<br />

Francisco, CA, Jossey-Bass, 55-70.<br />

Pintrich, P. R., (1989). The Dynamic Interplay of Student Motivation and Cognition in the College Classroom.<br />

In Ames, C. & Maehr, M. (Eds.), Advances in Motivation and Achievement: Vol. 6. Motivation Enhancing<br />

Environments, Greenwich, CT, JAI Press, 117-160,.<br />

Pintrich, P. R. & DeGroot, V. E., (1990). Motivational and Self-Regulated Learning Components of Classroom<br />

Academic Performance. Journal of Educational Psychology, 82 (1), 33-40.<br />

Pilkington, R. & Grierson, A., (1996). Generating explanation in a simulation-Based Learning Environment.<br />

International Journal of Human-Computer studies, 45 (5), 527-551.<br />

131


Rieber, L. P., (1990). Animation in Computer-Based Instruction. Educational Technology Research and<br />

Development, 38 (1), 77-86.<br />

Retalis, S. & Papasalours, A., (2005). Designing and Generating Educational Adaptive Hypermedia<br />

Applications, Educational Technology and Society, 8 (1), 69-79.<br />

Silverman, L. K., (2002). Upside-down Brilliance: The Visual-Spatial Learner, DeLeon Publishing, Denver, CO.<br />

Simonson, M., (1985). Persuasive Films: A Study of Techniques Used to Change Attitudes. Journal of Teaching<br />

and Learning Technologies. 1 (2), 39-48.<br />

Simonson, M. & Maushak, N., (1995). Situated Learning, Instructional Technology, and Attitude Change. In<br />

McLellan, H. (Ed)., Perspectives on Situated Learning, Cliffs, NJ, Englewood, 46-79.<br />

Stone, R., (2001). Virtual Reality for Interactive Training: An Industrial Practitioner’s View Point. International<br />

Journal of Human-Computer Studies. 55 (4), 699-711.<br />

Taylor, S. & Todd, P., (1995). Understanding Information Technology Usage: A Test of Competing Models.<br />

Information Systems Research. 6 (2), 144-176.<br />

Tong, A. K. Y. & Angelides, M. C. (2000) An Empirical Model for Tutoring Strategy Selection in Multimedia<br />

Tutoring Systems. Decision Support Systems. 29 (1), 31-45.<br />

Velayo, R. S., (2000). How do Presentation Modality and Strategy Use Influence Memory for Paired Concepts.<br />

Journal of Instructional Psychology. 27 (2), 126-133.<br />

Venkatesh, V., (1999). Creation of Favorable User Perceptions: Exploring the Role of Intrinsic Motivation. MIS<br />

Quarterly, 23 (2), 239-260.<br />

Weinstein, C. E. & Mayer, R. E., (1986). The Teaching of Learning Strategies. In Wittrock, M. (Ed.), Handbook<br />

of Research on Teaching, Macmillan, New York, NY, 315-327.<br />

Zimbardo, P. & Leippe, M., (1991). The Psychology of Attitude Change and Social Influence. Temple<br />

University Press, Philadelphia, PA.<br />

Zimmerman, B. & Pons, M., (1988). Construct Validation of a Strategy Model of Student Self-Regulated<br />

Learning. Journal of Educational Psychology, 80 (3), 284-290.<br />

132


Appendix A<br />

Questionnaire Validity and Reliability<br />

One measure of construct validity is the extent to which each item of a questionnaire correlates with the total<br />

score (Kerlinger, 1973). The correlation between the score for each item (question) and the total usefulness<br />

score was 0.65 to 0.84 (all significant at p0.05).<br />

We removed these two items from the subsequent analysis. Correlation between overall score for self-regulated<br />

learning and the 22 pertinent items (i.e., items 23–44 as depicted in Appendix B) showed that three items 26, 27<br />

and 37 were insignificant. Thus, these three items were removed from the subsequent analysis.<br />

Table 1A: Validity and Reliability of the Questionnaires<br />

Validity<br />

Reliability<br />

Questionnaire Correlation Coefficient<br />

Factor Analysis<br />

Cronbach alpha<br />

(all significant at alpha < 0.01) (Item communality range)<br />

Usefulness for Trial 1 0.65 – 0.84 0.84 – 0.88 0.83<br />

Usefulness for Trial 2 0.68 – 0.90 0.86 – 0.95 0.88<br />

Motivation 0.51 - .88 0.65 – 0.89 0.91<br />

Self-regulated learning 0.50 – 0.77 0.59 – 0.76 0.86<br />

The data were further analyzed using factor analysis (principal component analysis using Varimax rotation<br />

method with Kaiser normalization) to establish convergent and discriminant validity (Table 1A). The factor<br />

analysis for usefulness questionnaire scores in exam 3 and exam 4 showed one orthogonal factor with<br />

eigenvalues above 1.0 accounting for 97.18% and 99.18% respectively, with item communality ranging between<br />

0.84 and 0.95 (Table 2A). The factor analysis for “motivation and self-regulated learning” showed two<br />

orthogonal factors with eigenvalues above 1.0, together accounting for 76.4% of the variation, with item<br />

communality ranging between 0.59 and 0.89.<br />

The Cronbach alpha reliability score for the three questionnaires was 0.83 to 0.91 (Table 1A). These values are<br />

well within the thresholds suggested by Nunnally (1978). Thus, the questionnaires can be used as acceptable<br />

instruments for subsequent analysis.<br />

Table 2A: Factor Analysis for Perceived Usefulness<br />

Item Factor loading Item Communality<br />

Q1-Trial 1<br />

0.932<br />

0.880<br />

Q1-Trial 2<br />

0.932<br />

0.891<br />

Q2-Trial1<br />

0.897<br />

0.871<br />

Q2-Trial2<br />

0.876<br />

0.863<br />

Q3-Trial1<br />

0.898<br />

0.853<br />

Q3-Trial2<br />

0.917<br />

0.885<br />

Q4-Trial1<br />

0.896<br />

0.835<br />

Q4-Trial2<br />

0.946<br />

0.949<br />

Q5-Trial1<br />

0.911<br />

0.846<br />

Q5-Trial2<br />

0.946<br />

0.949<br />

Test for Random Assignment<br />

To test for random assignment of subjects to the two groups, the following analysis was performed. The<br />

ANOVA test was used to assess subject performance for the first and second multiple-choice exams (Table 4) to<br />

find out if there were any differences between the two groups regarding their domain knowledge. The result of<br />

the ANOVA test demonstrated no significant difference between the two groups: F = 1.104 (p = 0.299) for the<br />

first exam and F = 0.648 (p = 0.425) for the second exam. We also used ANOVA to test possible differences<br />

between the two groups in regard to the subjects’ “motivational beliefs” and “self-regulated learning strategies”.<br />

The results of ANOVA test showed no significant difference between the two groups: F = 0.079 (p = 0.971) for<br />

the motivational beliefs and F = 0.035 (p = 0.991) for the self regulated learning strategies.<br />

References<br />

Nunnally, J. C. (1978). Psychometric Theory, New York, NY: McGraw-Hill.<br />

133


Appendix B<br />

The following 44 items represent the Motivated Strategies for Learning Questionnaire (MSLQ) used in this<br />

study to measure students’ motivational beliefs and self-regulated learning. The number next to the items<br />

reflects the item’s actual position on the questionnaire.<br />

All the items were assessed on the following 7-point Likert-type scale:<br />

1 2 3 4 5 6 7<br />

not at all true of me |-------------|-------------|-------------|-------------|-------------|-------------|------------| Very true of me<br />

Motivational Beliefs<br />

2. Compared with other students in this class I expect to do well.<br />

6. I’m certain I can understand the ideas taught in this course.<br />

8. I expect to do very well in this class.<br />

9. Compared with other students in this class, I think I am good student.<br />

I am sure I can do an excellent job on the problems and tasks assigned for this class.<br />

13. I think I will receive a good grade in this class.<br />

16. My study skills are excellent compared with others in this class.<br />

Compared with other students in this class I think I know a great deal about the subject.<br />

19. I know that I will be able to learn the material for this class.<br />

1. I prefer class work that is challenging so I can learn new things.<br />

4. It is important for me to learn what is being taught in this class.<br />

5. I like what I am learning in this class.<br />

7. I think I will be able to use what I learn in this class in other classes.<br />

I often choose paper topics I will learn something from even if they require more work.<br />

14. Even when I do poorly on a test I try to learn from my mistakes.<br />

15. I think that what I am learning in this class is useful for me to know.<br />

17. I think that what we are learning in this class is interesting.<br />

21. Understanding this subject is important to me.<br />

3. I am so nervous during a test that I cannot remember facts I have learned.<br />

12. I have an uneasy, upset feeling when I take a test.<br />

20. I worry a great deal about test.<br />

22. When I take a test I think about how poorly I am doing.<br />

Self-Regulated Learning<br />

When I study for a test, I try to put together the information from class and from the book.<br />

24. When I do homework, I try to remember what the teacher said in class so I can answer the<br />

questions correctly.<br />

26. It is hard for me to decide what the main ideas are in what I read.<br />

28. When I study I put important ideas into my own words.<br />

I always try to understand what the teacher is saying even if it does not make sense.<br />

30. When I study for a test I try to remember as many facts as I can.<br />

31. When studying, I copy my notes over to help me remember material.<br />

When I study for a test I practice saying the important facts over and over to myself.<br />

I use what I have learned from old homework assignments and the textbook to do new assignments.<br />

39. When I am studying a topic, I try to make everything fit together.<br />

41. When I read material for this class, I say the words over and over to myself to help me<br />

remember.<br />

42. I outline the chapters in my book to help me study.<br />

When reading I try to connect the things I am reading about with what I already know.<br />

134


25. I ask myself questions to make sure I know the material I have been studying.<br />

27. When work is hard I either give up or study only the easy parts.<br />

32. I work on practice exercise and answer end of chapter questions even when I do not have to.<br />

Even when study materials are dull and uninteresting, I keep working until I finish.<br />

35. Before I begin studying I think about the things I will need to do to learn.<br />

37. I often find that I have been reading for class but do not know what it is all about.<br />

38. I find that when the teacher is talking I think of other things and do not really listen to what is<br />

being said.<br />

40. When I’m reading I stop once in a while and go over what I have read.<br />

43. I work hard to get a good grade even when I do not like a class.<br />

135


Appendix C<br />

A Sample of Exam Questions used in Trials 1 & 2<br />

End user resistance can be minimized through:<br />

A) Proper end user education and training.<br />

B) Improved communications with IS professionals.<br />

C) End user involvement in the development and implementation of new systems.<br />

D) All of the above.<br />

Management involvement in information systems can take the form of:<br />

A) Top management participation in an executive information technology committee.<br />

B) Management participation in steering committees for systems development projects.<br />

C) End user management participation in the systems development process and the management of<br />

information systems.<br />

D) All of the above.<br />

Your text defines change management as managing the process of implementing major changes in information<br />

technology, business processes, organizational structures, and job assignments to reduce the risks and costs of<br />

change, and optimize its benefits. A number of factors (dimensions) of change management are illustrated and<br />

discussed. Of these factors, which one is the major focus of organizational change management?<br />

A) Process<br />

B) People<br />

C) Technology.<br />

D) Knowledge management.<br />

Your text states that the E-business planning process ahs three major components - strategy development,<br />

resource management, and technology architecture. Strategy Development involves:<br />

A) Developing E-business and E-commerce strategies that support a company's E-business vision that<br />

focuses on customer and business value.<br />

B) Developing strategic plans for managing or outsourcing a company's IT resources.<br />

C) Developing strategic IT choices that reflect information technology architecture designed to support a<br />

company's E-business and E-commerce initiatives.<br />

D) None of the above applies to defining strategy development.<br />

Prototyping involves:<br />

A) The execution of the standard systems development cycle using CASE tools.<br />

B) A rapid generation of systems by information systems professionals without the need for end user input.<br />

C) The use of a fail-safe development process designed to ensure that an information system meets all user<br />

requirements without revision.<br />

D) An iterative and interactive development process with extensive end user involvement.<br />

Organizations are increasingly using encryption of data and firewall computers as methods to protect computer<br />

network resources. Firewalls are described as:<br />

A) The transmission of data through telecommunications lines in "scrambled" form.<br />

B) A "gatekeeper" system that protects a company's intranets and other computer networks from intrusion<br />

by providing a filter and safe transfer point for access to and from the Internet and other networks.<br />

C) The act of ensuring the accuracy, integrity, and safety of all E-business processes and resources.<br />

D) None of the above apply.<br />

A DBMS query language is designed to:<br />

A) Support information systems professionals in the development of complex application software.<br />

B) Support end users that wish to interrogate the database using English-like or natural language<br />

commands.<br />

C) Provide efficient processing of the database in the batch mode.<br />

D) Specify the content, relationships, and structure of a database.<br />

In the data planning and modeling process described in your text:<br />

A) Database development starts with a bottom-up data planning process.<br />

136


B) Subject area databases are consolidated to form the enterprise model.<br />

C) User views are generated by identifying the key data elements in the subject area databases, which are<br />

needed to perform specific business activities.<br />

D) The enterprise model is mapped directly into the physical database design.<br />

The logical structure of information in a database is contained in the<br />

A) Data dictionary.<br />

B) Data manipulation subsystem.<br />

C) Data administration subsystem.<br />

D) Data definition subsystem.<br />

A network that contains one or more host computers that provide some type of service to the other computers in<br />

a network is a:<br />

A) Peer-to-peer network<br />

B) Client/server network<br />

C) Local area network.<br />

D) Wide area network.<br />

137


Appendix D<br />

Perceived Usefulness Questionnaire<br />

All the items were assessed on the following 7-point Likert-type scale:<br />

Likely |________|________|________|________|________|________|________| Unlikely<br />

extremely quite slightly neither slightly quite extremely<br />

1. Using E-Tutor enables me to learn the MIS course materials more quickly<br />

2. Using E-Tutor improve my performance in the pertinent tests.<br />

3. Using E-Tutor enhance my learning effectiveness<br />

4. Using E-Tutor make it easier to learn the subject matter<br />

5. I find E-Tutor useful in learning subject matter of MIS course<br />

138


Chang, K.-E., Sung, Y.-T., & Hou, H.-T. (<strong>2006</strong>). Web-based Tools for Designing and Developing Teaching Materials for<br />

Integration of Information Technology into Instruction. Educational Technology & Society, 9 (4), 139-149.<br />

Web-based Tools for Designing and Developing Teaching Materials for<br />

Integration of Information Technology into Instruction<br />

Kuo-En Chang<br />

Department of Information and Computer Education, Center of Research for Educational Evaluation and<br />

Development, National Taiwan Normal University, Taipei, Taiwan<br />

Tel: +886223622841<br />

Fax: +886223512772<br />

kchang@ice.ntnu.edu.tw<br />

Yao-Ting Sung<br />

Department of Educational Psychology and Counseling, Center of Research for Educational Evaluation and<br />

Development, National Taiwan Normal University, Taipei, Taiwan<br />

Tel: +886223952445<br />

Fax: +886223413865<br />

sungtc@cc.ntnu.edu.tw<br />

Huei-Tse Hou<br />

Department of Information and Computer Education, National Taiwan Normal University, Taipei, Taiwan<br />

Tel: +886223622841<br />

Fax: +886223512772<br />

ho@ice.ntnu.edu.tw<br />

ABSTRACT<br />

Educational software for teachers is an important, yet usually ignored, link for integrating information<br />

technology into classroom instruction. This study builds a web-based teaching material design and<br />

development system. The process in the system is divided into four stages, analysis, design, development,<br />

and practice. Eight junior high school history teachers participated in the evaluation of the system. Through<br />

experts’ reviews and content analyses of their instructional materials and interviews, we found that<br />

instructional materials produced using the system appear to be more coherent and systematic, provide<br />

deeper and broader information for learning, apply more adequate teaching strategies, and lessen the design<br />

and development load on teachers.<br />

Keywords<br />

Integrated technology into instruction, web-based teaching material design and development, Instructional<br />

design<br />

Introduction<br />

The impact of information technology (includes web technology, software or hardware tools) on the overall<br />

teaching environment is becoming more pronounced. The ability to incorporate information technology into<br />

instruction has been deemed to be one of the most important professional competences of teachers (Fisher, 1997;<br />

Scheffler & Logan, 1999). Indeed, when used properly, information technology not only allows diverse ways for<br />

the presentation of materials (Chang, Sung, & Chiou, 2002; Roberts & Hsu, 2000), but also offers easy access to<br />

the wealth of resources available on the Internet and is useful for developing learner-centered strategies and<br />

activities (Chang, Chen, & Sung, 2001; Chang, Sung, & Lee, 2003; Linn, 2000).<br />

Compared to the pace of advancements in information technology, however, the rate of its adoption in the<br />

classroom has been slow. Studies reveal that teachers are often especially interested in such technology, but<br />

because of a lack of confidence, are intimidated by it (OTA, 1995; Willis, Thompson & Sadera, 1999).<br />

Researchers have identified several internal and external barriers to the adoption of technology, such as teachers’<br />

attitudes and willingness, their workload, the availability and accessibility of hardware and software, staff<br />

development, and institutional and technical support (Ertmer, 1999; Rogers, 2000).<br />

All these factors are interrelated. Lawson and Comber (1999), for instance, have argued that one of the key<br />

factors preventing teachers from applying information technology to teaching is their lack of computer literacy,<br />

which would require time to learn in addition to the time required for course preparation. However, Topper<br />

(1998) has observed that teacher training programs designed to facilitate technology literacy development often<br />

fail to achieve the desired results when teachers are over burdened with too much additional work. Cuban,<br />

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the<br />

copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies<br />

are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by<br />

others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior<br />

specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.<br />

139


Kirkpatrick, and Peck (2001) have also found that, although the increasing availability of computer equipment in<br />

schools offers easy access to computer resources, many teachers fail to alter existing patterns of teaching.<br />

The lack of software designed specifically for instruction might be one of the major reasons for the continued<br />

lack of willingness to use information technology. The strong emphasis on the learner’s perspective in recent<br />

years has led to many software products for teaching which focus on students’ learning needs instead of<br />

teachers’ instructional design facilitation. Hence, teachers themselves have benefited relatively little from such<br />

developments (Leu et al., 1998; Marx, Blumenfeld, Krajcik, & Soloway, 1998). The software for “integrating<br />

technology into instruction” is therefore needed by teachers to leverage information technology with minimal<br />

effort. Moreover, through such software, the burden of course preparation would be lightened, and thus teachers<br />

would be better motivated to apply information technology in their teaching. As long as they have the will, they<br />

are more likely to seek ways to integrate technology with teaching and try out new materials and methods<br />

(Norum, Grabinger, & Duffield, 1999). If the ultimate goal of integrating instruction with technology is to make<br />

the technology an integral part of how the classroom functions (International Society for Technology in<br />

Education, 2000), then the design and application of software for instructional purposes is an essential step.<br />

Most Web tools currently available for instructional design provide only assistance with website construction for<br />

either asynchronous learning or distance learning. For instance, WebCT (http://www.webct.com) and e-<br />

Education (http://www.jonesknowledge.com) both provide tools for teachers to construct an online classroom to<br />

place course content pages and to utilize other functions such as course schedule, discussion zone, online quiz,<br />

and bulletin board. However, these web-based tools fail to address the difficulty of finding needed information<br />

through Web searches, which limits the amount of supplementary material teachers can offer (Small, Sutton,<br />

Miwa, Urfels, & Eisenberg, 1998). Moreover, the emphasis of these web-based tools on distance learning is<br />

different from the functions emphasized in technology-incorporated teaching, such as the support of teaching<br />

strategies for classroom teaching.<br />

In short, a web-based teaching materials design and development system tailored to the needs of teachers who<br />

have a lower level of computer literacy is helpful for teachers wishing to incorporate technology into teaching. It<br />

is the goal of this study to develop a web-based environment that allows teachers to implement their instructional<br />

procedures, for example by developing teaching plans, preparing course materials, and managing teaching<br />

activities in an efficient and effective way. This system will focus more on each step of web-based teaching<br />

material design (and will highlight the steps of Analysis, Design, Develop and Practice), which offer a series of<br />

technological auxiliary tools for teaching. We will also look from the aspect of information-searching, providing<br />

URL management modules, offering keyword indexing, and record browsing history. In order to evaluate the<br />

system, eight junior high school history teachers were invited to participate in the evaluation. This paper uses<br />

experts’ reviews and content analyses of teachers’ instructional materials and interviews, and concludes that<br />

instructional materials produced by the system appear to be more coherent and systematic, provide deeper and<br />

broader information for learning, apply more adequate teaching strategies, and lessen the design and<br />

development load on teachers.<br />

System outline<br />

The system is divided into four stages: analysis, design, development, and practice. Table 1 shows the functions<br />

for each stage. The portal page URL of the system is http://elearning.ice.ntnu.edu.tw. Below is a detailed<br />

description of each.<br />

Analysis stage<br />

In this stage teachers analyze both learners’ characteristics and teaching content. Two functions are used for<br />

analyzing the progress and characteristics of learners. The first function is students’ basic information analyzer<br />

and manager, which records students’ basic information, including their past learning experience, family<br />

background, aptitudes, and what students feedback to teacher throughout the learning process. The second<br />

function is students’ score manager. Students’ score data accumulate over time with tests and exams as learning<br />

progresses. The score manager helps teachers better analyze students’ learning status.<br />

140


Stage Tasks<br />

Table 1. List of the functions for each stage<br />

Functions<br />

1. Analysis Learners’ characteristics analysis Students’ basic information analyzer and manager<br />

Students’ score manager<br />

Teaching material analysis Notepad<br />

Multimedia resource bank<br />

Web browser<br />

URL collector<br />

URL browser recorder<br />

URL resource bank<br />

2. Design Developing course outline Weekly course scheduler<br />

Syllabus builder<br />

Developing unit and lesson plans Unit plan builder<br />

Lesson plan builder<br />

3. Development Preparing teaching materials and Web Course page editor and manager<br />

pages such as online handouts, Personal webpage editor<br />

assignments, learning worksheets, Learning worksheet editor<br />

demos, references, supplementary Slide editor<br />

information, test questions, and slides for Test editor & manager<br />

classroom use.<br />

4. Practice Carrying out class teaching activities Daily course bulletin<br />

Online evaluation & communication Online synchronous communicator<br />

Online asynchronous communicator<br />

Online assignment evaluator<br />

With regard to teaching material analysis, teachers may utilize built-in databanks or conduct Web searches to<br />

collect information for the analysis and design of teaching materials. According to Keller (1983), multimedia<br />

presentations of materials usually increase cognition and arouse interest in learners. Therefore, the system<br />

provides a multimedia resource bank for easy retrieval by teachers. This databank contains many multimedia<br />

resources for teachers. Teachers can type a keywords related to the intended theme and relevant multimedia<br />

resources will then pop up for selection. Teachers may integrate these with the teaching materials they are<br />

developing.<br />

It is also noted that because of the complexity of Internet hyperlink connections, teachers often find themselves<br />

losing focus during Web searches (Park, 1991). In the other word, the hyperlinks around the Internet provide<br />

many different forms of various information. It is difficult for instructors and learners to search or even to use<br />

the resources in teaching or learning and it costs the learners too much time to filter the unrelated information.<br />

Furthermore, instructors and learners may perhaps get lost in many hyperlinks paths. To solve this problem, the<br />

system provides a notepad for teachers to record important information from the Web at any time during their<br />

online search. Small, Sutton, Miwa, Urfels, & Eisenberg (1998) pointed out that the chances of a teacher locating<br />

the right resources or adequate multimedia materials with just a few Web searches are slim. Hence, in addition to<br />

a web browser, the system provides teachers with a URL collector, a URL browser recorder, and a URL<br />

resource bank to assist teachers with online resource searching and gathering.<br />

The URL collector allows teachers to add new websites that they frequently use, and provides keyword search<br />

for URLs already stored in the system. The URL browser recorder helps record the names and URLs of websites<br />

teachers have visited using the system. The URL resource bank stores all relevant and assorted URLs. Teachers<br />

can easily access some other websites via Web links provided by the URLs in the databank and gather<br />

information for teaching materials design. The databank stores two types of URLs: (1) literature of educational<br />

or psychological theories; (2) course related information, such as lesson plans and examples of teaching activity<br />

design.<br />

Design stage<br />

During this stage, teachers produce their teaching outline and plan based on the analyses in the previous stage.<br />

Teachers often draw on past experience and personal beliefs about teaching as they teach (Mannaz, 1999).<br />

Personal styles and beliefs are also reflected in their choice of educational software (Niederhauser & Stoddart,<br />

141


2001). The system thus aims to assist teachers to produce outlines and plans in their own style. The model also<br />

provides a teaching plan and drafting tools to facilitate collation of lesson plans and self-reflection upon past<br />

teaching performances.<br />

There are four tools used in the design stage. Weekly course scheduler provides an academic calendar and<br />

monthly planner to facilitate time management. Syllabus builder assists teachers in developing, editing, and<br />

managing syllabuses, which is a course plan for the whole semester. Unit plan builder assists teachers in<br />

developing, editing, and managing unit plans, each of which is a teaching plan for a specific unit. Lesson plan<br />

builder assists teachers in developing, editing, and managing daily lesson plans, and includes a memo feature for<br />

particular lessons or classes.<br />

Development stage<br />

During this stage teachers prepare supplementary materials according to the draft teaching plan prepared in the<br />

design stage and the actual teaching needs, and then place the completed materials onto the website for students’<br />

review before and after class. Most of the available tools for web-based teaching material development and<br />

course management lack the needed support for instructional webpage production and fail to provide assistance<br />

with the preparation of supplementary teaching materials such as learning worksheets and slides. The key task in<br />

this stage is therefore to produce all the materials (learning worksheets, slides, etc.) necessary for instructional<br />

webpage production and teaching activity implementation in a web-based support environment. To familiarize<br />

teachers with the new operating environment, the system incorporates the standard Microsoft Office editing tools<br />

(MS Office 2000), which the majority of teachers should be familiar with. The tools in this stage are as follows:<br />

1. Course page editor and manager: This tool provides teachers with an environment for instructional webpage<br />

editing and combines real-time on-line multimedia functions to facilitate the production of instructional<br />

webpages.<br />

2. Personal webpage editor: This tool assists teachers in developing their own personal teaching pages through<br />

which students can access the course pages designed by the teacher.<br />

3. Learning worksheet editor: This tool helps teachers develop learning worksheets which are becoming more<br />

popular with teachers.<br />

4. Slide editor: This tool incorporates functions from Microsoft PowerPoint and assists teachers with slide<br />

preparation.<br />

5. Test editor and manager: This tool incorporates functions from Microsoft Word and assists teachers with<br />

test question editing. It also helps produce a test template according to the type and number of questions<br />

teachers need in order to reduce the extra work with test paper formatting and layout.<br />

Practice stage<br />

During this stage teachers utilize the Internet to conduct various classroom and after hour teaching activities and<br />

employ online tools for distance discussion and evaluation. In addition to synchronous and asynchronous<br />

network communication tools, the system also helps teachers evaluate students’ papers online. The following<br />

four functions are provided by this stage.<br />

1. Daily course bulletin: This tool provides an online bulletin for announcements and course daily messages to<br />

students.<br />

2. Online synchronous communicator: This function provides a discussion forum similar to online chat rooms<br />

to allow teachers to provide counseling or respond to course related inquiries.<br />

3. Online asynchronous communicator: This function provides an asynchronous discussion zone similar to a<br />

message board to allow teachers and students to discuss course related problems.<br />

4. Online assignment evaluator: This function allows teachers to evaluate and score students’ papers and<br />

assignments on-line.<br />

Experiment<br />

Participants<br />

Eight history teachers from three junior high schools in Taipei participated in this study. The participants have<br />

one to nine years of teaching experience with an average of six years. They had a basic understanding of<br />

computer operations such as word processing and surfing the Internet, but had no experience in the use of webbased<br />

applications for the design or development of teaching materials.<br />

142


Research design<br />

The eight participating teachers were randomly divided into an experimental group (referred to as Teachers A, B,<br />

C, and D) and a control group (Teachers E, F, G, and H). The subjects were requested to develop their teaching<br />

materials for an assigned teaching unit. The experimental group was given the system, and the control group an<br />

Internet browser and word processor.<br />

Tools and materials<br />

Course Content Material: The participants of this study were given the unit “Domestic Politics of the Ming<br />

Dynasty” from the “Glories of the Late Imperial Era” section of the junior high school history book. The<br />

participants were assigned the text of the unit along with the relevant parts in the teacher’s handbook as a<br />

reference for designing the teaching materials and developing four hour teaching periods.<br />

Scale for evaluating instructional materials: In order to analyze the differences between the materials developed<br />

by the experimental and control groups, we developed a five-point Likert-type scale (1=strongly disagree to<br />

5=strongly agree). Zero points were given when a desired item was missing from the teacher’s materials. The<br />

scale comprises three categories of eight items (Table 2), including the quality of supplementary information<br />

gathered in the analysis stage, the teaching plan developed in the design stage, and the teaching materials<br />

produced in the development stage.<br />

Questionnaire for interview: To obtain the opinions of the participating teachers in the experimental group, the<br />

researcher developed a set of 21 structured questions for the interview. The questions are divided into several<br />

categories such as background, opinions about the system interface, functions, guiding mechanism, system<br />

usefulness, and the personal process of instructional design.<br />

Procedures<br />

The procedures of the experimental group were implemented in three stages: (1) Giving instructions and<br />

background interview (60 min.). During this stage, a teacher’s general background is probed during the<br />

interview, including his or her past experience with the use of computers in teaching. The proposed system is<br />

also explained to the teacher. (2) Planning, data searching, and preparing worksheet. This stage starts with the<br />

actual analysis and design done by the participants (80 min.). Based on the textbook and the teacher’s handbook,<br />

as well as information gathered from the Web, teachers then proceeded to produce the “teaching plan,” the<br />

“lesson plan,” the “learning worksheet,” and the “detailed activity plan” with the help of the system tools. The<br />

researcher also conducted interviews with the participants regarding this particular stage of the experiment (30<br />

min.). (3) Developing teaching materials as Web pages. This stage starts with the participants executing the<br />

development stage (80 min.). The participants decide on the content for their Web pages, which they may<br />

develop with the system tools. After developing teaching materials, the researcher interviewed the participants in<br />

this stage for 30 minutes.<br />

For the control group, due to a lack of Web page production experience, it was difficult for the participants to<br />

implement Web pages with other Web page tools like FrontPage. So, they were asked to complete the analysis<br />

and design stages. The data in the development stage was not included in the comparison. This part of the<br />

experiment was thus divided into the following two stages: (1) Giving instructions and background interview (60<br />

min.). The content of the interviews was the same as for the experimental group. The functions of Microsoft<br />

Word and Internet Explorer were also explained to the teachers. (2) Planning, data search, and preparation for<br />

worksheet. During this stage teachers conducted their own analysis and design of the teaching unit (80 min.).<br />

This includes the course content analysis, the use of computers to prepare and save supplementary materials<br />

(using Word or Notepad), formulating a teaching plan, and finally, developing learning worksheets. After the<br />

teachers completed their work, the researcher then conducted interviews on this part of the experiment (30 min.).<br />

Results<br />

The finished instructional materials were evaluated by two junior high history teachers with 22 and 27 years of<br />

experience. The materials used were the teaching plans, supplementary materials, and work sheets. The<br />

evaluation results for eight teachers are shown in Table 3. The Spearman’s rank correlation coefficient indicates<br />

143


a strong agreement between the two experts’ ratings on the quality of the materials, ρ =1.00 (p


offer criticisms. The teachers in the control group choose lecturing as their main strategy, with only key lecture<br />

points of the unit listed on their teaching plans. No other methods are mentioned in their teaching plans, nor were<br />

worksheets developed to supplement the lecture.<br />

Coordination, coherence, and creativity among teaching plans and materials<br />

With respect to the coordination between teaching plan and teaching materials, as well as the coherence of the<br />

teaching materials, the experts’ ratings for the two groups show a marginal significance (U = 2, p = 0.057). This<br />

suggests that the experimental group shows greater coordination between the supplementary information<br />

collected, activity plans, learning worksheets, and Web pages developed using the system. For overall creativity<br />

of teaching materials, experts’ ratings show a statistically significant difference (U = 0, p = 0.014), indicating<br />

that the experimental group demonstrated more creativity than the control group when given the same amount of<br />

preparation time and teaching content.<br />

Table 5 lists the supplementary materials, the content of the teaching plan and learning worksheet, and the major<br />

headings of the Web page produced by teacher B. As the table shows, teacher B adopted direct lecturing and<br />

guided discussion as his main strategy in the teaching plan, which was developed by incorporating the<br />

information gathered from the Web with the textbook content (e.g., a unit about Chen-Ho’s voyage to the west in<br />

the textbook and the materials about Chen-Ho on the Web). The order and content of the lecture are closely<br />

related to the supplementary material the teacher has prepared. Also, teacher B’s learning worksheet is<br />

significantly related to the supplementary material and the progress of discussion and lecture. For example,<br />

teacher B used the question “Did Chen Ho’s boats reach Africa?” which is an issue currently being debated by<br />

historians. Other teachers in the experimental group showed similar features with their instructional materials.<br />

The control group not only failed to produce learning worksheets, but the information they gathered show no<br />

direct relation to the lecture outline in their teaching plans. As can be seen from Table 4, the data collected on<br />

“public flogging at court with clubs,” a unique punishment during the Ming Dynasty, by teacher F are absent<br />

from the key points in his outline. The supplementary material provided by teacher F is not organized for the<br />

purpose of teaching, but rather is presented to students in its original form, which is complicated and somewhat<br />

inconsistent with the entire teaching plan.<br />

Table 4. Information collected and planned teaching strategies by participants<br />

Teacher Information gathered from the Web Activities or strategies suggested in the teaching<br />

plan and worksheets<br />

A 1. Six relevant pictures.<br />

2. Supplementary information on the maritime expeditions<br />

of Cheng Ho (his life story, stories of his voyages,<br />

relevant websites, historical sites and information left<br />

from his journeys).<br />

3. Introductory information on Islam.<br />

B 1. Six relevant pictures.<br />

2. Stories about the founder of the Ming Dynasty, Tai-tsu,<br />

on his founding of a new dynasty, political contributions,<br />

and problematic policies.<br />

3. Supplementary information on the Jing Nan Rebellion<br />

and the contributions of the succeeding Emperor Chen<br />

after the rebellion.<br />

4. Supplementary information on the reforms by Chief<br />

Minister Chang Chu-cheng.<br />

5. Information on Tung-lin Partisan Disputes and the<br />

analysis of causes for the decline of the Ming Dynasty.<br />

6. Supplementary information, relevant websites, and<br />

archeological materials on Cheng Ho’s voyages west.<br />

C 1. Four relevant pictures.<br />

2. Supplementary information on Cheng Ho’s voyages to<br />

the West (his life story, uniqueness of his expeditions).<br />

D 1. Two relevant pictures.<br />

2. The background of the founding Emperor Chu Yuanchang,<br />

his rise and important measures at the beginning<br />

Uses lectures as the main thread and supplements<br />

it with a self-designed learning worksheet, and<br />

employs a light, everyday theme of “Overseas<br />

Tour Guide of the Ming Dynasty” to introduce the<br />

supplementary materials of the unit.<br />

Uses lectures as the main strategy, coupled with<br />

much elaborate and broad-ranging supplementary<br />

information. Prior to the introduction of the theme<br />

of the lesson, the teacher poses proper questions to<br />

elicit students’ responses and stimulate their<br />

thinking as part of his effort to raise students’<br />

motivation. The learning worksheet provides<br />

students with questions that help them to review<br />

the lesson, and to think and reason.<br />

Uses lectures as the main thread and supplements<br />

it with a map in the learning worksheet. The<br />

teacher attempts to help students brainstorm for<br />

ideas regarding Cheng Ho’s maritime voyages by<br />

comparing the two great geographical discoveries<br />

of the East and West in human history.<br />

Uses lectures as the main thread and allows<br />

students to express their own views and judgments<br />

about historical figures before the conclusion of<br />

145


of the dynasty.<br />

3. Information on “public flogging at court with clubs.”<br />

4. Introductory websites on Ming Founder Tai-tsu.<br />

E 1. One picture of Cheng Ho.<br />

2. Brief introduction to Cheng Ho’s life.<br />

3. Brief introduction to the Ming Dynasty and a table of<br />

Ming emperors.<br />

F 1. Relevant information on “public flogging at court with<br />

clubs.”<br />

G 1. One picture of Cheng Ho.<br />

2. Supplementary information on Cheng Ho’s maritime<br />

expeditions (his life story and the uniqueness of his<br />

voyages).<br />

H 1. Information on the founding of a dynasty by Tai-tsu and<br />

his most important aid Liu Chi.<br />

the lesson, and then introduces various judgments<br />

by historians in order to offer students a broader<br />

perspective. The learning worksheet also requests<br />

students evaluate various historical figures.<br />

Listing the headings for major talking points in the<br />

teaching plan, no notes on any classroom<br />

activities.<br />

Listing the headings for major talking points in the<br />

teaching plan, no notes on any classroom activities<br />

or learning worksheet to supplement the lecture.<br />

Listing the headings for major talking points in the<br />

teaching plan, no notes on any classroom activities<br />

or learning worksheet to supplement the lecture.<br />

Listing the headings for major talking points in the<br />

teaching plan, no notes on any classroom activities<br />

or learning worksheet to supplement the lecture.<br />

Table 5. Content analysis of teacher B’s instructional materials<br />

Instructional materials Titles of main items<br />

Supplementary information 1. Pictures related to the teaching material.<br />

collected from the Web 2. Stories about the founding of the Ming Dynasty by Emperor Tai-tsu.<br />

3. Tai-tsu’s contributions and failures.<br />

4. Jing Nan Rebellion and Emperor Chen’s achievements.<br />

5. Information on the reforms of Chief Minister Chang Chu-cheng.<br />

6. Analysis of causes of the decline of the Ming Dynasty.<br />

7. Information on the Tung-lin Partisan Disputes.<br />

8. General and archeological information on Cheng Ho’s voyages to the West.<br />

Teaching plan<br />

1. Based on the knowledge about emperors already learned, discuss what<br />

(The teacher uses lecture and family background or life experience it takes to become an emperor.<br />

guided discussion as the main 2. Ask students if they can name any peasant-turned emperors and if they have<br />

strategy in his teaching plan; heard about Chu Yuan-chang.<br />

the right column shows the 3. Introduce Chu Yuan-chang.<br />

outlines for the lecture and 4. Discuss whether his background as a former peasant will make any<br />

discussion.)<br />

difference in the policies he makes and the regime he forms.<br />

5. Introduce various political measures during Tai-tsu’s rule.<br />

6. Discuss the Jing Nan Rebellion.<br />

7. Discuss the possible political measures Emperor Chen would take to retain<br />

power after he overthrew his predecessor.<br />

8. Tell the class that the maritime industry was highly developed in the Ming<br />

Dynasty and ask if they have heard of Cheng Ho’s expeditions.<br />

9. Discuss the places Cheng Ho visited.<br />

10. Lecture on the savior of the Ming Dynasty—Chang Chu-cheng.<br />

11. Lecture on the causes and impact of the decline of the Ming Dynasty.<br />

Learning worksheet<br />

1. Who was the founding emperor of the Ming Dynasty? Name two peasant-<br />

(The learning worksheet is turned emperors in the history of China.<br />

developed both for classroom 2. What are the three reforms that Tai-tsu instituted to ensure a better life for<br />

and after-class use with an aim his people? What are the potential problems with these measures? Give two<br />

to help students think and examples.<br />

practice; the right column 3. What emperor was pulled off the throne during the Jing Nan Rebellion? And<br />

shows the questions listed on who succeeded him?<br />

the sheet.)<br />

4. Who contributed to the maritime expansion in the Ming Dynasty and thus<br />

secured China’ status in Southeast Asia?<br />

5. Did Ming’s sovereignty ever reach Africa?<br />

6. What are the most important economic measures of Chief Minister Chang<br />

Chu-cheng?<br />

7. What are the important causes of Ming’s downfall? When did it officially<br />

end? Who is the last emperor that committed suicide?<br />

Teaching material page 1. Opening up a new prospect—Chu Yuan-chang<br />

146


(The headings and titles on the<br />

teaching material page are<br />

shown on the right, which<br />

includes the pictures and URL<br />

hyperlinks collected by the<br />

teacher for students’ reference.)<br />

Analysis of interview content<br />

2. Important measures at the beginning of the new dynasty<br />

3. Cheng Ho’s voyages to the West<br />

4. Chief Minister Chang Chu-cheng and his “integrated taxation” scheme<br />

5. History champion (This section contains challenging questions that the<br />

teacher poses on the Web.)<br />

The summarization and analysis of the interviews with participating teachers reveal information about<br />

experiences and feelings caused by using different media for teaching material design. All four teachers in the<br />

experimental group showed satisfaction with the friendly environment provided by the proposed system, and<br />

agreed that the web editor and notepad are easy to use, thereby saving time during Web search. Regarding the<br />

notepad, the teachers indicated that everything returned from any single search could be easily retrieved later for<br />

the production of various instructional materials. As teacher A put it,<br />

“…it’s quite a handy tool for editing. It’s just like writing your own lesson plans or reorganizing<br />

various elements. In fact, the moment you start using the notepad, you feel its handiness…I never<br />

thought it would be this easy because in the past when I used the Internet, I used to open a window for<br />

Word on the side so that I could cut and paste along the way. Usually I had to do a lot of dragging and<br />

pasting. After I have searched for a while I would tell myself that it is time to start editing…which I<br />

usually spent an awful lot of time on, so by the time I finished editing and tried to get back to where I<br />

left off searching, boy!...Where was I? What else is left for me to supplement with extra material? So I<br />

find this tool really saves me a lot of time.”<br />

Teachers also consider the URL databank helpful and convenient to their online search effort. As teacher C<br />

comments,<br />

“…I think in terms of teaching resource collection. This tool is more convenient because the<br />

information in the databank is already sorted and so contains more relevant websites. That’s why it is<br />

convenient. You can save a lot of time with this databank because you no longer need to weed out<br />

totally irrelevant websites that pop up during your Web search. For example when you type in a few<br />

keywords, sometimes it returns over 200 results, many of them mixed with English and Chinese. It is<br />

very frustrating….”<br />

Teachers further indicated that the proposed system helps them compare different units, which, in turn, promotes<br />

coherent and integrated thinking, offers inspiration, and increases creativity for teaching material design during<br />

their Web search. During their interview, teachers B and D related<br />

Teacher B: “…one of its strengths is that you can store a series of notepads…and retrieve them later,<br />

which makes comparing and connecting units easier.”<br />

Teacher D: “…when gathering information becomes easier, you usually get more inspiration….”<br />

The four teachers in the control group admitted during their interviews that searching the Web for teaching<br />

resources is no easy task. One of the difficulties cited is that the search engines available often return many links<br />

that are totally irrelevant to what is needed. As a result they have to spend a lot of time screening the links one<br />

by one and sometimes have to visit a website to determine if the information is indeed useful. Many times their<br />

search returns with information that is totally unexpected. Teachers in this group also agreed that computer<br />

technology can provide a complement to conventional written texts because it allows concurrent display of text<br />

and graphics.<br />

Discussion and conclusions<br />

The role of information technology is becoming more important in the instructional environment. However, the<br />

lack of appropriate software for instructional purposes inhibits the process of integrating technology with<br />

instruction. To compensate for this lack, we developed a web-based teaching material design and development<br />

system and tested its effectiveness in helping in-service teachers develop their teaching plans and materials. We<br />

hope that the facilities and friendly user interface of the system will be able to reduce teachers’ workloads and<br />

increase the quality of teaching plans and materials designed on the Internet.<br />

147


Comparing works designed using the system with those designed using an Internet browser and word processor,<br />

we have some encouraging findings. For example, previous studies (Small et al., 1998) indicated that seeking<br />

suitable materials from the Internet is time consuming and frustrating owing to the messiness of the databank,<br />

which then limits the amount of supplementary materials teachers can offer. We addressed this difficulty by<br />

providing a URL collector, a URL browser recorder, and a URL resource bank to assist teachers. Teachers in the<br />

experimental group all successfully and without much difficulty completed many instructional materials with the<br />

supporting functions. On the contrary, using the same of preparation time, teachers in the control group failed to<br />

produce instructional materials of similar quality. Furthermore, Roberts et al. (2000) found that teaching<br />

products made through computer-based technology were superior in design and required less time than<br />

handmade ones, but that there was no significant difference in the creativity of the two kinds of products.<br />

However, teachers in the experimental group not only completed better instructional materials, but also their<br />

materials were more creative and diverse, and the overall plans and materials were more coordinated and better<br />

structured.<br />

Innovative instruction relies on both a teacher’s willingness to engage in designing innovative teaching plans and<br />

on having access to facilities for helping achieve such innovation. Computer technology is one of the major tools<br />

that should provide facilities for innovation. However, increased access to equipment and training neither led to<br />

widespread teacher use nor altered existing patterns of teaching practice (Cuban, 2001; Havista & Lesgold,<br />

1996). One possible reason for the failure to achieve the desired level of innovation despite the investment in<br />

effort and resources is the lack of suitable software to optimize the functions of equipment and training. Only<br />

when teachers believe that their workload can be reduced by using computers and quality of teaching can be<br />

enhanced will they be willing to increase their use of computers for instruction. The results of this study also<br />

support this assertion. Both the quantitative and qualitative data from our interviews demonstrate that teachers<br />

using our system experienced a lower workload and higher efficiency. The result was that they then generated<br />

more ideas and designed more activities reflecting their individual styles and originality.<br />

We believe that the design of software for instructional purposes and the exposure to such software are the most<br />

direct ways to link equipment and training, and to optimize the possible benefits of using information<br />

technology. The design and application of our system may serve as a reference for related studies. Due to time<br />

constraints and environmental factors however, this study did not conduct an observation and evaluation of the<br />

use of the system in a real teaching context. Future studies will investigate the usefulness of the system in a<br />

classroom context and other issues related to teaching material design on the Internet.<br />

Acknowledgement<br />

This research was supported by a grant from the National Science Council, Republic of China, under contract<br />

number NSC93-2524-S-003-014.<br />

References<br />

Chang, K. E., Chen, S. F. & Sung, T. C. (2001) Learning through computer-based concept mapping with<br />

scaffolding aid. Journal of Computer-Assisted Learning, 17 (1), 21-33.<br />

Chang, K. E., Sung, Y. T. & Chiou, S. K. (2002). Use of hierarchical hyper concept map in Web-based courses.<br />

Journal of Educational Computing Research, 27 (4), 333-351.<br />

Chang, K. E., Sung, Y. T. & Lee, C. L. (2003). Web-based collaborative inquiry learning system. Journal of<br />

Computer Assisted Learning, 19 (1), 56-69.<br />

Cuban, L., Kirkpatrick, H. & Peck, C. (2001). High access and low use of technologies in high school<br />

classrooms: Explaining an apparent paradox. American Educational Research Journal, 38, 813-834.<br />

Ertmer, P. A. (1999). Addressing first- and second-order barriers to change: Strategies for technology<br />

integration. Educational Technology Research and Development, 47, 47-61.<br />

Fisher, M. M. (1997). The voice of experience: In-service teacher technology competency recommendations for<br />

pre-service teacher preparation programs. Journal of Technology and Teacher Education, 5, 139-147.<br />

148


Havista, N. & Lesgold, A. (1996). Situational effects in classroom technology implementations: Unfulfilled<br />

expectations and unexpected outcomes. In Kerr, S. T. (Ed.), Technology and the future of schooling: Ninety-fifth<br />

Yearbook of the National Society for the Study of Education, part 2 (131-171). Chicago, University of Chicago<br />

Press.<br />

International Society for Technology in Education (2000). National Educational Technology Standards for<br />

Students: Connecting Curriculum and Technology. O. R. Eugene: Author.<br />

Keller, J. M. (1983). Motivational design and instruction. In Reiguluth, C. M. (Ed.), Instructional design theories<br />

and models: An overview of their current status (383-434). Hillsdale, NY: Lawrence Erlbaum Associates.<br />

Lawson, T. & Comber, C. (1999). Superhighway technology: Personnel factors leading to successful integration<br />

of information and communications technology in schools and colleges. Journal of Information Technology for<br />

Teacher Education, 8, 41-53.<br />

Leu, D. J., Hillinger, M., Loseby, P., Balcom, M., Dinkin, J., Eckels, M., Johnson, J., Mathews, K. & Raegler, R.<br />

(1998). Grounding the design of new technologies for literacy and learning in teachers’ instructional needs. In<br />

Reinking, D., McKenna, M., Labbo, L. D. & Kieffer, R. (Eds.), Handbook of literacy and technology:<br />

Transformations in a post-typographic world (203-220). Mahwah, NJ: Erlbaum.<br />

Linn, M. C. (2000). Designing the knowledge integration environment. International Journal of Science<br />

Education, 22, 781-796.<br />

Mannaz, M. (1999). An expert teacher’s thinking and teaching and instructional design models and principles:<br />

An Ethnographic study. Educational Technology Research and Development, 46, 37-64.<br />

Marx, R. W., Blumenfeld, P. C., Krajcik, J. S. & Soloway. E. (1998). New technologies for teacher professional<br />

development. Teaching and Teacher Education, 14, 33-52.<br />

Niederhauser, D. S. & Stoddart, T. (2001). Teachers’ instructional perspectives and use of educational software.<br />

Teaching and Teacher Education, 17, 15-31.<br />

Norum, K., Grabinger, R. S. & Duffield, J. A.(1999). Healing the universe is an inside job: Teacher’s views on<br />

integrating technology. Journal of Technology and Teacher Education, 7, 187-203.<br />

OTA (Office of Technology Assessment). (1995). Teachers and technology: Making the connection.<br />

Washington, DC: Government Printing Office.<br />

Park, O. (1991). Hypermedia: Functional features and powerful ideas. NewYork: Basic Books.<br />

Roberts, S. K. & Hsu, Y. S. (2000). The tools of teacher education: Preservice teachers’ use of technology to<br />

create instructional materials. Journal of Technology and Teacher Education, 8, 133-152.<br />

Rogers, P. A. (2000). Barriers to adopting emerging technologies in education. Journal of Educational<br />

Computing Research, 22, 455-472.<br />

Scheffler, F. L. & Logan, J. P. (1999). Computer technology in schools: What teachers should know and be able<br />

to do. Journal of Research on Computing in Education, 31, 305-326.<br />

Siegel, S., & Castellan, N. J. (1988). NonparametricSstatistics for the Behavioral Sciences. New York: McGraw<br />

Hill.<br />

Small, R. V., Sutton, S., Miwa, M., Urfels, C. & Eisenberg, M. (1998). Information seeking for instructional<br />

planning: An exploratory study. Journal of Research on Computing in Education, 31, 204-215.<br />

Topper, A. (1998). Co-constructing practices of teaching with technology: Working collaboratively with TE<br />

instructors on technology adoption. Technology and Teacher Education Annual 1998, 74-748.<br />

Willis, J., Thempson, A., & Sadera, W. (1999). Research on technology and teacher education: Current status<br />

and future directions. Educational Technology Research and Development, 47, 29-45.<br />

149


Tselios, N., Stoica, A., Maragoudakis, M., Avouris, N., & Komis, V. (<strong>2006</strong>). Enhancing user support in open problem<br />

solving environments through Bayesian Network inference techniques. Educational Technology & Society, 9 (4), 150-165.<br />

Enhancing user support in open problem solving environments through<br />

Bayesian Network inference techniques<br />

Nikolaos Tselios<br />

Human-Computer Interaction Group, Electrical and Computer Engineering Dept and ICT in Education Group,<br />

Dept of Educational Sciences and Early Childhood Education, University of Patras, Greece<br />

nitse@ee.upatras.gr<br />

Adrian Stoica<br />

Human-Computer Interaction Group, Electrical and Computer Engineering Dept, University of Patras, Greece<br />

stoica@ee.upatras.gr<br />

Manolis Maragoudakis<br />

Wireless Communication Laboratory, Electrical and Computer Engineering Dept, University of Patras, Greece<br />

mmarag@wc.ee.upatras.gr<br />

Nikolaos Avouris<br />

Human-Computer Interaction Group, Electrical and Computer Engineering Dept, University of Patras, Greece<br />

avouris@ee.upatras.gr<br />

Vassilis Komis<br />

ICT in Education Group, Dept of Educ Sciences and Early Childhood Education, University of Patras, Greece<br />

komis@upatras.gr<br />

ABSTRACT<br />

During the last years, development of open learning environments that support effectively their users has<br />

been a challenge for the research community of educational technologies. The open interactive nature of<br />

these environments results in users experiencing difficulties in coping with the plethora of available<br />

functions, especially during their initial efforts to use the system. In addition, -from the tutors’ perspective-<br />

the problem solving strategies of the students are often particularly difficult to identify. In this paper, we<br />

argue that such problems could be tackled using machine learning techniques such as Bayesian Networks.<br />

We show how we can take advantage of log files obtained during field studies to build an adaptive help<br />

system providing the most useful support to the student, according to the state of interaction. On the other<br />

hand, we attempt to support the tutor, by automating the process of diagnosing students’ problem solving<br />

strategies using Bayesian Networks. The presented approaches are discussed through examples of two<br />

prototypes that have been developed and corresponding evaluation studies. These studies have shown that<br />

the proposed approach can effectively support the tasks of students and tutors in such open learning<br />

environments.<br />

Keywords<br />

Bayesian Belief Networks, Open problem solving environments, Inference algorithms, On-line adaptation,<br />

Adaptive help, Automated problem solving strategy identification.<br />

Introduction<br />

During the last few years, a number of open problem-solving environments have been built that are based on the<br />

constructivist approach. Open problem solving environments are computer-based learning environments that let<br />

students actively explore certain concepts while they are engaged in problem solving, with emphasis in the<br />

active, subjective and constructive character of learning (Luger and Stubblefield, 1998). Compared to traditional<br />

learning environments the student’s activity cannot be reduced in a sequence of pre-defined tasks. In general,<br />

through an open problem solving environment the student is engaged in processes such as model building,<br />

investigation and reflection. As a result, students have the freedom to explore various entities in their own unique<br />

way. The open nature of these environments, results in users’ activity taking various forms and the patterns of<br />

use of the tools that are included in them not to be fully anticipated. Therefore, the complexity of such<br />

environments and the open nature of the tasks can often lead to poor usability, which can be an obstacle to obtain<br />

the expected pedagogical value from their use.<br />

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the<br />

copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies<br />

are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by<br />

others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior<br />

specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.<br />

150


In general, through an open problem solving environment the student is engaged in processes such as model<br />

building, investigation and reflection. The process of building such a model is usually carried out in three phases.<br />

First, the system allows users to express their ideas, through ‘entities’ that are related to objects, corresponding<br />

to their phenomenological status. These objects have properties, directly manipulated by the students, and<br />

behaviour. At the second level, the students can relate explicitly or implicitly a number of entities to create a<br />

more abstract entity, considered also as a ‘construct’, depicting an object from a group of uniform objects, that<br />

take meaning in the context of a phenomenon, system, process or conjecture (Komis et al., 2001,<br />

Dimitracopoulou and Komis, 2005). Finally, the students are able to ‘run’ the model to validate their hypothesis<br />

and observe the function of their model, often aided by data visualisation tools such as equation plots, graphs, etc<br />

(Komis et al., 2002).<br />

Improving usability of these environments is an objective that could be achieved in various ways, such as.<br />

applying user centred design techniques (Norman, 1986) at design time or by building user support components<br />

for run time. User centred design propose advanced usability evaluation techniques during system design. For<br />

instance, various usability evaluation techniques that take into account both the pedagogical value and the<br />

usability of the environment have been proposed (Squires and Preece, 1999, Avouris et al., 2000). Furthermore,<br />

more complex user task modelling approaches have been proposed for the design of such environments (Tselios<br />

et al., 2002). On the other hand, user support at run time through adaptive systems is an approach that could<br />

improve usability. It should be recognized that, overall, artificial intelligence techniques have not succeeded to<br />

deliver the expected results in the educational field in the form of Intelligent Tutoring Systems, despite the<br />

existence of some sporadic promising results (Woolf et al., 2001, see also Kinshuk and Russell, 2002, for a<br />

classification of adaptable and adaptive systems). However, the premise of adaptive system behaviour through<br />

which higher usability and increased system transparency can be obtained remains a valid scientific objective.<br />

Our approach is compatible with the vision of Suthers et al. (2001) which stress the value of ‘Minimalist AI’ in<br />

education: Instead of trying to build smart machines that teach -a rather optimistic goal demanding a great effort<br />

of modeling knowledge in a particular subject, as well as pedagogy strategies and explanation mechanisms-, this<br />

approach suggests providing machines with abilities to ‘respond to the semantics of student activities and<br />

constructions, to test the educational value of these abilities, and add functionality as needed to address<br />

deficiencies in the utility of the system’.<br />

In the research reported in this paper, we attempt to investigate the usefulness of Bayesian Networks in tackling<br />

two significant problems, common in open learning environments. These two problems are complementary in<br />

nature and address difficulties of students and tutors when engaged in activities in open learning environments:<br />

First, from a student perspective, an adaptive help module is presented which automatically recognizes the<br />

most probable next action and presents relevant help items. The need for such an adaptive help system is<br />

significant, especially in the case of an open problem solving environment where the cognitive effort to deal<br />

with the task is affected by concerns on how to handle the various tools that may be used for accomplishing<br />

a certain task.<br />

From a tutor perspective, we attempt to present a practical method to classify effectively problem solving<br />

strategies expressed by pupils while they were using such an open problem solving environment. Due to the<br />

nature of such environments, problem solving strategies could be numerous. Posteriori analysis of log data<br />

in order to identify the strategy expressed by the students could be a rather tedious and painstaking process.<br />

Thus, a method to automatically classify the solutions presented to the users could substantially increase the<br />

evaluation of the learning process.<br />

In the presented cases, we try to implement a state-based AI approach instead of a rich knowledge-based<br />

(Nathan, 1998). We attempt to identify important states, features and repeatable patterns in the students’ system<br />

interaction cycle in order to infer about their intentions or adopted strategy during their problem solving activity,<br />

while retaining at the same time the sense of self-activity and engagement. Bayesian networks are used as<br />

modelling tools in this state based approach in both cases.<br />

This paper is organized as follows: First, we present an overview of interesting applications of Bayesian Belief<br />

Networks (BBN) in various problems in the field of educational technology. Then, we briefly present Bayesian<br />

probabilistic theory together with a description of the notion of BBN’s and their implementation in educational<br />

environments. Next, we present our approach to build an adaptive help system in the ModelsCreator open<br />

problem solving environment. The proposed approach of adapting user support through a BBN is described. The<br />

BBN was built using a large amount of log files of actual usage of the system. Next, a system testing and<br />

evaluation experiment that took place in order to demonstrate the benefits of the proposed architecture is<br />

presented, followed by a short presentation of the results obtained from our experience of student modelling with<br />

BBN in open problem solving environments. Finally, we present our approach to automatically identify users’<br />

151


strategies during their interactions with an open learning environment that is used for teaching geometrical<br />

concepts to pupils aged 11-14 years. In the proposed approach, a BBN has been used to infer problem-solving<br />

strategies that the pupils applied in order to solve a given problem from logs of activity of typical pupils. In this<br />

case, identification of user’s strategies was done with the aim to facilitate classification of the solutions presented<br />

by the students and aid the evaluation of the learning process, a typical tutor task.<br />

Use of Bayesian Belief Networks in Educational Systems<br />

In recent years, symbolic modelling approaches applied in traditional AI problems, have been replaced by<br />

implicit models built from rich data sets through techniques proposed by Machine Learning and Knowledge<br />

Discovery from typical data of a specific domain. These techniques have produced efficient algorithms during<br />

the last years and found new areas of application. Among them, a technique that has certain advantages and has<br />

received attention during the last few years is based on Bayesian Belief Networks (BBN’s, Niedermayer, 1998).<br />

A Bayesian Belief Network (Stephenson, 2000) is a directed acyclic graph, where each node represents a random<br />

variable of interest and each edge represents direct correlations between the variables.<br />

Bayesian reasoning is based on formal probability theory and is used extensively in several current areas of<br />

research, including pattern recognition, decision support and classification. Assuming a random sampling of<br />

events, Bayesian theory supports the calculation of more complex probabilities from previously known results<br />

(Luger and Stubblefield, 1998). The advantages of BBNs include the simple process for constructing<br />

probabilistic networks even from a relatively limited amount of data, the efficient algorithms to evaluate degrees<br />

of belief for instances of a node and the versatile knowledge representation which such networks provide.<br />

Due to the underlying probabilistic model that describes the belief on the existence of a specific event, BBNs are<br />

considered one of the most effective ways to represent uncertainty. By formulating a limited, approximate but<br />

representative cognitive model of the users suitable for the specific problem, and then modeling uncertainty in<br />

human computer interaction, future activity could be supported. This is done through the modeling of a user’s<br />

activity, recorded in click streams of typical user behavior when interacting with a software system. This could<br />

lead to interpretation of the nature of the cognitive processes involved and to more efficient ways to support<br />

future users’ tasks. This approach is particularly suitable for learning applications, in which interactions are<br />

complex and support is often needed.<br />

So in recent years, various prototypes have been produced, demonstrating that BBNs are suitable for effective<br />

modelling of student behaviour (Jameson et al., 1995). BBNs have been utilized in various ways to achieve<br />

adaptability in educational environments in terms of determining student goals, determining feedback,<br />

curriculum sequencing and fine-tuning of the pedagogical strategy to deliver knowledge. ANDES (Conati et al.,<br />

1997), is a system that teaches physics problem solving techniques to college students, uses BBNs to identify the<br />

current problem solving approach of the user. ANDES also use a BBN to determine what hints to provide to the<br />

user by identifying how the student is solving a problem and how he has progressed down the solution path.<br />

Extensions of the modeling process to cope with issues that arose in scaling up the model to a full-scale, field<br />

evaluated application coupled with results of several evaluations of Andes which provide evidence on the<br />

accuracy of the implemented models are presented by Conati et al., (2002). Bunt and Conati (2003) attempt a<br />

generalization to the student modelling process in open problem environments. They argue in favour of using<br />

such a technique, finding it beneficial even for those learners who do not already possess the skills relevant to<br />

effective exploration. The scope of their model is twofold: To assess the effectiveness of a learner's exploratory<br />

behaviour in an open learning environment and to monitor the learners' actions in the environment unobtrusively<br />

in order to maintain the unrestricted nature of the interaction is one of the key assets of such environments.<br />

In addition to reasoning at a lower interaction level about student actions it is possible to use a BBN in order to<br />

reason at a higher level. Collins et al. (1996) constructed a BBN that represents a hierarchy of skills for<br />

arithmetic, containing the test questions, the theory and the user’s skill in specific topics. The semantics implied<br />

by the links, represent which question refers to a specific topic. The system considers the current estimate of the<br />

student’s ability to compute the probability of responding correctly to each question contained in the database<br />

schema. Then the system selects an item that the student has a near to 50% chance of answering correctly. With<br />

regard to the selected pedagogical approach, the criteria of item selection could be altered appropriately. CAPIT,<br />

(Mayo and Mitrovic, 2000), is a constraint-based tutor for English capitalization and punctuation that uses BBN<br />

for long term student modeling. An evaluation of this system showed that a group of students using the adaptive<br />

version learned the domain rules at a faster rate than the group that used the non-normative version of the same<br />

system.<br />

152


Usage of Bayesian Networks is not limited to monitoring the student’s behaviour in an educational environment.<br />

For example, Vomlel (2004) stresses the use of Bayesian Networks for educational testing. He presented a series<br />

of case studies referring to operations that use fractions. He showed that a Bayesian network models relations<br />

between required skills to carry out these operations and improve the process of student's evaluation. Also Xenos<br />

(2004), presents a methodological approach based on Bayesian Networks for modelling the behaviour of the<br />

students of a bachelor course in computers in an Open University that deploys distance educational methods.<br />

This method offered an effective way to model past experience, which can significantly aid decision-making<br />

regarding the educational procedure. According to Xenos (2004) this approach can also be used for assessment<br />

purposes regarding the current state enabling tutors to identify best and worst practices.<br />

As discussed in this section, probabilistic models, such as BBN have focused on traditional “drill and test”<br />

systems to a great extent. However, in the research reported here, we attempt to show that applications of this<br />

method are not necessarily constrained only in the aforementioned type of systems. In the following, we show<br />

applications of BBNs in open learning environments with a much more complex and non linear nature of user<br />

system interaction. As described in the rest of the paper, this class of educational systems could also benefit from<br />

a BBN approach, in a variety of ways.<br />

Bayesian Modeling<br />

A Bayesian Belief Network (BBN) is a significant knowledge representation and reasoning tool, under<br />

conditions of uncertainty (Mitchell, 1997). Given a set of variables D = , where each variable Xi<br />

could take values from a set Val(Xi), a BBN describes the probability distribution over this set of variables. We<br />

use capital letters as X,Y to denote variables and lower case letters as x,y to denote values taken by these<br />

variables. Formally, a BBN is an annotated directed acyclic graph (DAG) that encodes a joint probability<br />

distribution. We denote a network B as a pair B=, (Pearl, 1988) where G is a DAG whose nodes<br />

symbolize the variables of D, and Θ refers to the set of parameters that quantifies the network. G embeds the<br />

following conditional independence assumption:<br />

Each variable Xi is independent of its non-descendants given its parents.<br />

Θ includes information about the probability distribution of a value xi of a variable Xi, given the values of its<br />

immediate predecessors. The unique joint probability distribution over that a network B describes<br />

can be computed using:<br />

Learning BBN from data<br />

P ( X ... X ) =<br />

B<br />

1<br />

N<br />

N<br />

∏<br />

i=<br />

1<br />

P(<br />

x | parents ( X ))<br />

i<br />

The process of efficiently detecting the most suitable network is not straightforward. Thus, a BBN should be<br />

learned from the training data provided. Learning a BBN unifies two processes: learning the graphical structure<br />

and learning the parameters Θ for that structure. In order to seek out the optimal parameters for a given corpus of<br />

complete data, we directly use the empirical conditional frequencies extracted from the data (Cooper and<br />

Herskovits, 1992). The selection of the variables that will constitute the data set is of great significance, since the<br />

number of possible networks that could describe N variables equals to:<br />

2<br />

N ( N −1)<br />

2<br />

where N is the number of variables (Jeffreys, 1939).We use the following equation along with Bayes theorem to<br />

determine the relation r (or Bayes factor) of two candidate networks B1 and B2 respectively:<br />

where:<br />

P(<br />

B1<br />

| D)<br />

r = (1)<br />

P(<br />

B | D)<br />

2<br />

P(<br />

D|<br />

B)<br />

P(<br />

B)<br />

P ( B|<br />

D)<br />

= (2)<br />

P(<br />

D)<br />

i<br />

153


P(B|D) is the probability of a network B given data D.<br />

P(D|B) is the probability the network gives to data D.<br />

P(D) is the ‘general’ probability of data.<br />

P(B) is the probability of the network before seen the data.<br />

Applying equation (1) to (2), we get:<br />

P(<br />

D | B1)<br />

P(<br />

B1)<br />

r = (3)<br />

P(<br />

D | B ) P(<br />

B )<br />

2<br />

Having not seen the data, no prior knowledge is obtainable and thus no straightforward method of computing<br />

P(B1) and P(B2) is feasible. A common way to deal with this is to assume that every network has the same<br />

probability with all the others, so equation (3) becomes:<br />

P(<br />

D | B1)<br />

r =<br />

P(<br />

D | B )<br />

2<br />

The probability the model gives to the data can be extracted using the following formula (Glymour and Cooper,<br />

1999):<br />

P(<br />

D | B)<br />

=<br />

n qi<br />

ri<br />

ijk<br />

qi<br />

ri<br />

qi<br />

∏<br />

i=<br />

1 j=<br />

1 Ξ<br />

k=<br />

1 Ξ<br />

Γ(<br />

+ Nij<br />

) Γ(<br />

)<br />

qi<br />

ri<br />

qi<br />

∏∏<br />

Ξ<br />

Γ(<br />

)<br />

where:<br />

Γ is the gamma function.<br />

n equals to the number of variables.<br />

ri denotes the number of values in i:th variable.<br />

qi denotes the number of possible different value combinations the parent variables can take.<br />

Nij depicts the number of rows in data that have j:th value combinations for parents of i:th variable.<br />

Nijk corresponds to the number of rows that have k:th value for the i:th variable and which also have j:th<br />

value combinations for parents of i:th variable.<br />

Ξ is the equivalent sample size, a parameter that determines how readily we change our beliefs about the<br />

quantitative nature of dependencies when we see the data. In our study, we follow a simple choice inspired<br />

by Jeffreys (1939) prior. Ξ is equal to the average number of values variables have, divided by 2.<br />

Given the great number of possible networks produced by the learning process, a search algorithm has to be<br />

applied. We follow greedy search with one modification: instead of comparing all candidate networks, we<br />

consider investigating the set that resembles the best current model the most.<br />

In general, a BBN is capable of computing the probability distribution for any partial subset of variables, given<br />

the values or distributions of any subset of the remaining variables. Note that the values have to be discretised,<br />

and different discretisation size affects the network. As we shall discuss in the result section, BBNs constitute a<br />

significant tool for knowledge representation, visualising the relationships between features and subsets of them.<br />

This fact has a significant result on identifying which features actually affect the class variable, thus reducing<br />

training data size without any significant impact in the performance.<br />

However, we should stress the fact that transforming the Bayesian Belief Network elicitation process to a<br />

tractable procedure is not enough. The most important step, while attempting to resolve a problem using the<br />

application of such a technique, is the establishment of a deep understanding of the nature of the problem. This<br />

understanding could lead to a manageable transformation of the problem domain, where applications of BBNs<br />

could serve as an important inference tool. From our expertise obtained dealing with the adaptation problems<br />

described in the following, attempting to re-express the problem to be modelled should first include an initial<br />

understanding and clarification of the prominent variables of the domain which influence our reasoning about the<br />

state or the goal of the student while executing a task. These variables, coupled with nodes indicating the issue<br />

(the ‘class’ variable) that is needed to infer upon, are used to feed the BBN construction process. The states of<br />

the variables represent actual user’s actions, neither at a high task level, nor in the lowest keystroke level of<br />

interaction. The granularity of the abstraction is heavily depended by the nature of the problem and our aim. It<br />

2<br />

Γ(<br />

Ξ<br />

+ N<br />

)<br />

154


seems that different problems may need a slightly different approach. For example, while attempting to<br />

recognise a student’s current goal, the temporal order of her recent actions significantly influences our inference<br />

about her forthcoming action. When we attempt to infer general goals of the user from log files, it seems that<br />

focusing upon the frequency of a tool’s usage is a preferable selection.<br />

Using BBN to Construct an Adaptive Help System<br />

This section presents an adaptive help system that was built using BBNs, for ModelsCreator (MC), an open<br />

learning environment for modelling activities of young students. ModelsCreator allows users to create models<br />

representing aspects and phenomena of the natural world and provides them with tools for testing their models<br />

(simulations, graphs, tables of values, etc.). ModelsCreator design and implementation has been inspired by two<br />

main design principles: (a) support of expression through different kinds of reasoning in a simplified and<br />

synthetic mode and (b) model mechanisms that derive from different subject matters, which permit<br />

interdisciplinary approaches (Komis et al., 2001, Dimitrakopoulou and Komis, 2005). Models can be built in<br />

MC, using qualitative reasoning, as well as quantitative and semi-quantitative relations that can be used to link<br />

various objects, representing primitive concepts. The system supports a high level of visualisation, combines<br />

modeling tools with simulations and incorporates alternative and multiple forms of representation (Von<br />

Glasersfeld, 1987). Visual representations of entities in a specific topic (i.e. food, water, photosynthesis)<br />

constitute essential elements of the environment since they have the mediating role of providing students with a<br />

‘concrete’ framework to reason on an abstract level (Figure 1).<br />

.<br />

Concerning the MC user interaction design, a direct manipulation space, suitable for young students has been<br />

designed. A view of this space is shown in Figure 1, in which the modelling environment of MC can be seen. It<br />

has been shown (Komis et al., 2002, Ergazaki et al., 2005) that this system can provide a rich and constructive<br />

learning experience to young students. For example, in Figure 1, in the main modelling space a typical model of<br />

a biological phenomenon (photosynthesis) is shown. Entities like a plant, the sun, the soil, the leaf of the plant<br />

and the substances of the air are included in the model and are related through qualitative relations. On the left, a<br />

library of entities is included. These can be dragged into the modelling space. On the right, the available relations<br />

are shown. Finally, on the top, various tools for model manipulation, for creating various representations of the<br />

model and running the model are included.<br />

Figure 1. ModelsCreator’s modelling environment<br />

155


Online help adaptation using Bayesian networks<br />

ModelsCreator contains a rich help system. However, it was discovered that in its earlier form this help system<br />

was particularly difficult to use by young students who had to spend a long time browsing in large areas of<br />

information in order to receive support. So a BBN has been developed in the context of the reported research,<br />

used in order to adapt the user support (help) system of ModelsCreator. The BBN was built using data collected<br />

from typical real world user interaction with the MC environment. The model was based on the selected data, so<br />

the approach we followed can be considered as a data-centric one. The data used to train the structure and the<br />

probabilities of the Bayesian network was obtained from a quite large amount of log files produced by<br />

ModelsCreator during previous experiments, involving 15 students aged 11-14 years. . Representative tasks were<br />

given to the students to accomplish. An extract of the log files used can be seen in Figure 2. This is a history of<br />

the sequence of operations of a user in the MC environment. Each operation might have associated attributes so<br />

the first operation of the user was an insertion of an entity in the modelling space. The attribute of this specific<br />

action was the kind of entity that was inserted (the “Plant” entity).<br />

A log file pre-processing procedure took place and resulted in the building of a database containing some 2200<br />

logged actions. Our effort was to construct a BBN in order to obtain a belief about the next most probable action<br />

of the student with respect to the current state of the interaction flow. We assumed that by predicting the next<br />

most probable operation, we can infer the user intention and thus relate the most relevant section of the help<br />

system if the user sought support. The database that was built in this pre-processing phase contained the required<br />

fields to predict the next action given the two previous actions. The field Property of the action has been used for<br />

each action describing the particular nature of the action. For example, when a user selects a particular attribute<br />

of an Object, the selected attribute is recorded in the property field. The fields of the database are: Previous<br />

Action (Paction), Previous Property of the carried out Action (Pprop), Previous Action Time, Current Action<br />

(Caction), Current Action Property (Cproperties), Current Action Time, Next Action (Naction), Next Property<br />

and the Time difference between current and previous (CPTime_d). The database has this structure since the<br />

temporal ordering of the actions has not been taken into consideration.<br />

The user activity in MC is very unpredictable. Therefore, there is virtually no constraint in solutions that the user<br />

might adopt and express during problem solving. Thus, an adaptation procedure in a higher cognitive goal level<br />

(e.g., concerning specific goals which the user attempts to accomplish, or even a sub-goal as a part of a pupil’s<br />

approach to solve a problem) could prove very difficult to be carried out. Instead, the proposed approach focuses<br />

on general patterns that can occur in the stream of user interaction with the software.<br />

The log file extract shows a representative student’s interaction with the system (Gudzial, 1993). First the user<br />

attempts to enter three entities (Plant, Leaf, Sun) to the MC’s working space (Actions 1,3,4). Subsequently, she<br />

selects specific attributes for each entity during her effort to build a model describing causal-effect relationships<br />

between the members of the model (Actions 2 and 5-12). Finally she executes the model in order to observe the<br />

behaviour of the model. To further examine the exact nature of the relationships between the entities, she creates<br />

a barchart representing the evolution of the three selected entity’s attributes over time (Actions 13-18).<br />

# Time Action Property of the action<br />

01. 00 : 08 : 14 InsertEntity Plant<br />

02. 00 : 08 : 19 ChooseAttribute Growth->Plant<br />

03. 00 : 08 : 37 InsertEntity Leaf<br />

04. 00 : 08 : 40 InsertEntity Sun<br />

05. 00 : 08 : 46 ChooseAttribute Photosynthesis->Leaf<br />

06. 00 : 08 : 49 ChooseAttribute Intensity of Light->Sun<br />

07. 00 : 09 : 07 InsertRelation Proportional<br />

08. 00 : 09 : 07 ConnectRelation Photosynthesis<br />

09. 00 : 09 : 07 ConnectRelation Intensity of Light<br />

10. 00 : 09 : 48 InsertRelation Proportional<br />

11. 00 : 09 : 48 ConnectRelation Growth<br />

12. 00 : 09 : 48 ConnectRelation Photosynthesis<br />

13. 00 : 10 : 00 RunModel<br />

14. 00 : 11 : 19 BarChartActivation<br />

15. 00 : 11 : 21 GraphChooseAttribute Growth->Plant<br />

16. 00 : 11 : 23 GraphChooseAttribute Intensity of Light->Sun<br />

17. 00 : 11 : 24 GraphChooseAttribute Photosynthesis->Leaf<br />

18. 00 : 11 : 31 RunModel<br />

Figure 2. Extract of the log file used to build the BBN<br />

156


In the derived BBN (Figure 3), causal relationships between Previous action (PAction) and Current action<br />

(CAction), and between Current action and Next action have been depicted. From an interaction flow<br />

perspective, the structure is semantically meaningful and the relation between the previous and current action of<br />

a student is an expectable empirical model of interaction. To find the probability of occurrence of an instance of<br />

the next action variable, only the current instance of the current action is needed. The structure of our Bayesian<br />

network is a tree structure denoting that it is suitable to be used even in real time training and adaptation as well,<br />

since the time required to obtain such a structure from data is polynomial (Stephenson, 2000).<br />

Figure 3. The final Bayesian network obtained. Previous action (PAction), Current action (Caction) and the time<br />

difference between them (CPTime_d), influences network’s belief concerning the most probable next action<br />

Environment architecture modification<br />

The architecture of ModelsCreator was enriched with a new module that incorporated the developed BBN and<br />

enhances the functionality of the User Support Module (Help System). The architecture of the new module<br />

named Adaptive User Support Module (AUSM) is conceptually presented in Figure 4.<br />

The role of the new module is twofold. First, it is to collect data from the user’s stream of actions and to write<br />

them into a database. Subsequently, to identify the current interaction state using the collected data and to<br />

provide at any time the most appropriate help topics according to the probabilistic estimation made using the<br />

developed BBN. The database obtained is used as input for a Bayesian modelling tool that will process the data<br />

in the database and will construct the BBN that models our problem. The resulting network is exported in Hugin<br />

Lite file format. The module parses the file that contains the network and uses the conditional probability table<br />

within to provide the most useful help topics to ModelsCreator by predicting the most probable actions of the<br />

student to follow. This Bayesian network is actually used by a module that interacts with our open problemsolving<br />

environment, ModelsCreator and with a Bayesian modelling tool through files (it reads the BBN<br />

description file and writes the database with the data to update the probabilities of the network).<br />

F1<br />

Topics (T)<br />

T<br />

Topic<br />

Provider<br />

BBN<br />

A<br />

Action (A)<br />

Action<br />

Tracker<br />

Figure 4. Data exchange between modules in the frame of the Enhanced User Support System of ModelsCreator<br />

At run time the developed module operates as follows: Every action of the user is sent to the AUSM module<br />

through an appropriate method call. When the user asks for help the ModelsCreator will call another method of<br />

A<br />

Training<br />

A<br />

DB<br />

157


AUSM that will return the most probable topics for the current state of the interaction. These actions are<br />

obtained by instantiating in the Bayesian network the current action node and calculating the probabilities for the<br />

next action instances. Each next action is related to a relevant help topic. When a new action is performed the<br />

AUSM module writes in the database the data relating to the previous, current and next actions. Afterwards, it<br />

sends the action performed further so that the module is able to instantiate the current action node with the<br />

current action performed thus providing the adaptive version of user support.<br />

An example of the actual behaviour of the AUSM module is showed in Figure 5. In order to connect two entities<br />

with each other, the student has to enter those entities into the main space, select at least an attribute in each<br />

entity, drag a relationship and finally connect the edges of the relationship with the desired attributes, by pointing<br />

exactly at the area of the entity where the verbal representation of the attribute is presented. This procedure posed<br />

significant problems to the students, and patterns of interaction showing repeated attempts to properly connect a<br />

relationship were detected in the log files obtained. When the user asks for help, while she tries unsuccessfully to<br />

connect two objects with a relation, MC’s extended version presents the help section proposing three help topics<br />

suggested as relevant to the current interaction state. All help topics are presented in a task driven form to help<br />

the user carry out specific tasks. That is, the help system is not constrained to a description of each element of<br />

the user interface which is of limited help to the user, but it describes in detail how a representative task is<br />

carried out using MC’s environment. In the specific example described, the three proposed help items explain<br />

how to connect a relation (which is actually the desired help item), how to run a model and how to insert a<br />

relation into the MC’s environment (Figure 5). This approach considerably facilitates the task of finding the<br />

desired help item: Instead of selecting from a plethora of available help topics concerning a variety of actions,<br />

the user is provided with the most probable three. If the desired help item is not successfully suggested by the<br />

AUSM as in this example, the system alternatively permits viewing the help topics, sorted in alphabetical order<br />

and grouped by subject.<br />

Module evaluation<br />

Evaluation of the developed system was carried out in three different ways. First, the predictive performance has<br />

been measured using the training data set. Secondly, through inspection of the efficiency of the realised AUSM<br />

system against log files obtained by another study, and finally with a user observation involving actual users<br />

while executing representative tasks with the system.<br />

User support<br />

window<br />

Figure 5.The user interface of the adaptive user support environment. Three options are provided to the user in<br />

the help window, with the most probable one actually shown to the user<br />

158


It is common practice to evaluate BBN predictive performance by testing the network against data that were not<br />

used for training it. This is an advantage of such data-driven approach against efficiency-centric and expertcentric<br />

approaches for development of an adaptive system (Heckerman, 1996). The effectiveness of the Bayesian<br />

Network obtained was validated by using the 10-fold cross validation method. With this method, the data<br />

population is randomly divided to training and validation data sets in a 9:1 ratio, repeated ten times. The mean<br />

performance of the developed network was found to be 88,43% (+-1,36%, p


direct experimentation with a new software environment has been recorded also by other researchers (Hellman,<br />

1989).<br />

Some specific observations of the user study are discussed next. In some cases, (for example (a) activity of<br />

linking two objects in a model with a relation and (b) the selection of variables to create a graph), the provided<br />

help was found to be critical for successful and uninterrupted task execution. This was discovered during the<br />

user study. These problems were attributed to the fundamental gap between the designer’s conceptual model<br />

(actual system design) and the user’s conceptual model (expectations of the users referring to the actual way to<br />

carry out specific actions). On another occasion, the users experienced frustration when they repeatedly tried to<br />

change the direction of a specific relation between two objects, an action that is not directly supported by the<br />

MC. Finally, the users found the cut and paste tools and functions difficult to use. Concerning the AUSM’s<br />

presentation of information, they found that the explanations are clear and well written, but they would like to<br />

have more links to relative topics in the body of the help text.<br />

Finally, the actions carried out during the usability evaluation of the AUSM were recorded, collected and<br />

analyzed. They were analyzed against the relevance of the three help items presented to the user, concerning the<br />

actual user’s next action. The results were in line with the preliminary evaluation of AUSM. The AUSM<br />

presented first the desired help item in the 40,181% of analysed actions, as second in the 27,765% of actions and<br />

as third most likely in the 5,869% of cases, with a total forecast success 73,815%. The analysis of the events<br />

seems to be in line with the user’s subjective evaluation of the AUSM.<br />

Using Bayesian Networks for automatic classification of problem solving strategies<br />

In this section, development of a tool for automatic classification of problem solving strategies is described. The<br />

source data for BBN construction were captured from user observation sessions of 30 high school students while<br />

using the C.AR.ME microworld. This environment is an open interactive system using multiple representations<br />

for learning concepts of geometry, i.e. equivalent shapes and surface measure methods, (Kordaki and Potari,<br />

1998). The experiment concerned activities for solving two distinct problems: (a) transformation of a nonconvex<br />

polygon into a geometrically equivalent shape and (b) surface comparison of a non convex polygon with<br />

a square. Students were asked to solve these two problems with all the possible approaches that might find<br />

appropriate, using the tools offered by the microworld. The analysis of the problem solving strategies expressed<br />

by the students is extensively described in the work of Kordaki and Potari, (1998). The different problem<br />

solution strategies that the students used were classified in thirteen (13) categories. The most common strategy<br />

was described as “Using the measurement function”, that was applied by 23.7% of the students, while the “Using<br />

the automatic measurement function” was applied by 18.4% . These two strategies involved direct use of<br />

provided tools. Other strategies involved direct manipulation of shapes and combination of tools with direct<br />

manipulation and transformation of the drawn shapes.<br />

114 log files have recorded the sequences of operations of the 30 students to solve the given problems. Each one<br />

of them was related to a specific problem solving strategy. These log files were used to construct the BBN. In<br />

most of these log files, extensive comments were associated, describing the strategy expressed by the student,<br />

accompanied by a screenshot showing the final state of the microworld, at the end of the problem solving<br />

process. These log files were analysed using the Usability Analyzer (Tselios et al., 2002), a tool supporting<br />

analysis of log files captured from educational environments. In this tool, the evaluator can review the<br />

interaction process, by observing simultaneously the captured log files, comments taken during the field study<br />

and recorded screenshots showing the student’s interaction with the educational software.<br />

The next phase was to store the log files’ information into a table. The fields used were the 20 different low level<br />

student operations recorded and the frequency of usage for each one in every problem solving strategy. For<br />

example, Cut and Paste operations depict editing actions, MeasuringAreas depicts usage of a tool to measure the<br />

area of a specific area, and Triangle, or Rectangle operations depicts usage of automated tools to transform a<br />

geometric shape to another type of equivalent area. The hypothesis used to build the BBN and transform<br />

accordingly the interaction data, was the hypothesis that whenever a student adopts a specific strategy to solve a<br />

problem, he carries out a set of specific operations with high frequency. This kind of activity could be traced in<br />

her log file, and could help us classify automatically the problem solving strategy. Additionally information<br />

stored, referred to the total number of interaction events (sum variable) and the type of strategy used as<br />

recognized originally by the evaluator (class variable, see Figure 6).<br />

160


ΒΒΝ Construction<br />

The diagrammatic representation of the BBN obtained is shown in Figure 6. In general, a BBN is able to derive<br />

the probability distribution for any given variable subset, for specific values of the remaining variables. From the<br />

structure of the network it is shown that the startdraw and enddraw variables -which are the events created when<br />

the student begins or completes a shape’s drawing accordingly- are depending the one from the other, but are not<br />

connected with the main body of the network. This is not something unexpected, since these events indicate only<br />

the start and the end of the user’s activity, but not the exact nature of the specific problem solving strategy. In<br />

addition, four (4) more types of operations used in the log files, UnitIteration, Erase, Clear and Symmetry are<br />

found to be independent from any other variable and they were excluded from the graph.<br />

Figure 6. Bayesian network obtained to infer for the problem solving strategy expressed by the student from the<br />

log files<br />

Useful information can be extracted by observing the BBN graph. An example of the use of the network<br />

obtained is presented in Figure 7. For specific relative high usage of triangle and square transformations<br />

(Triangle and Square variables have been set to specific values, see Figure 7) the probability distribution in the<br />

‘class’ variable heavily suggests that the most probable problem solving strategy followed by the student is the<br />

C7, i.e. the one that involves usage of automatic measurement function (60% to 9.3% for the second most<br />

probable strategy, C5).<br />

Comparison with other methods<br />

In order to validate the effectiveness of the network, that is the ability to correctly predict the problem solving<br />

strategy originally expressed by the student, the ten fold cross validation method has been used. The<br />

performance of the BBN was 84.07% (+- 6.75% for p


Figure 7. An example of use of the obtained BBN. For given values of variables ‘Rectangle’ and ‘Triangle’ the<br />

probability the used followed the strategy C7 is very high (above right)<br />

Table 2. Performance comparison of categorization techniques<br />

Method Bayesian networks Naive Bayes Decision Tables C4.5<br />

% success<br />

rate<br />

84,07% 72,56% 63,71% 64,60%<br />

As one can deduct from the above table, Bayesian networks outperform all other methodologies, both of similar<br />

nature (such as the statistical-based Naïve Bayes which does not take into account any dependencies between the<br />

variables) and the decision-tree based (such as C4.5 and Decision Tables). Regarding the former case, Bayesian<br />

networks alleviate the over-restrictive assumption on the conditional independency on the attributes of the naïve<br />

Bayesian classifier, which is also often unrealistic. Bayesian networks, can reason under conditions of<br />

uncertainty, taking the semantic interrelation of attributes into account. Concerning the decision-tree based<br />

algorithms, we could argue that information gain, a metric that is the core of the algorithms can provide a clear<br />

view on the value of each attribute but can also lead to large trees, in which the task of pruning some redundant<br />

nodes is difficult and may cause the stall of the algorithm in local maxima. On the other hand, Bayesian<br />

networks are based on the probability of a structure over the given training set, which may be an NP-hard<br />

problem but using intelligent search strategies can overcome the problem of local maxima.<br />

Conclusions<br />

This paper presents the application of Bayesian Networks techniques for tackling two important problems in the<br />

frame of two open problem solving environments (MC and CARME), with very promising results. In the first<br />

case, we discussed the design and implementation of the adaptive user support module AUSM. The performance<br />

of the developed module during the evaluation experiment was very promising. The system, when fed with<br />

interaction data of a new problem solving task was able to correctly predict 3 out of 4 cases for the next user<br />

162


action and therefore to potentially provide the user with useful support in an efficient way. The proposed<br />

approach serves as a scaffolding mechanism during modelling activities. The support system helps students<br />

accomplish a complex task requiring specific actions, at critical points of their task. The proposed scaffolding<br />

mechanism, realised with the use of Bayesian Networks, does not reduce the transparency of the system. Instead,<br />

it provides the ability to the student to move ahead over potentially critical points in terms of tools operations. In<br />

the scaffolding process adopted, the system provides help to the students on those elements of tasks that are<br />

beyond their capacity, and allow them to concentrate upon the elements that are within their range of<br />

competence. This is accomplished with identification of frequent interaction patterns using Bayesian Networks<br />

and subsequent correlation with desired task accomplishment, thus enabling the system to infer the desired help<br />

item. It provides help just when it is needed thus reducing the effort to search for relevant support, while<br />

maintaining a focus on the modelling process without interruption and disorientation.<br />

In the second case of automatic classification of problem solving strategies in an open problem solving<br />

environment, the approach produced good results. The classification rate of the BBN algorithm is very high in<br />

comparison with other popular algorithms, even with a small amount of data provided. However, the BBN could<br />

be proven unsuccessful to recognize strategies with a low occurrence rate represented in the reference sample. A<br />

future research goal is to estimate the lowest possible sample for each category related to the total, which leads<br />

to unbiased and solid results. Various techniques have been proposed in the literature for tackling this problem<br />

(e.g. Daskalaki et al. <strong>2006</strong>), These could be particularly useful in our case, since often rare problem solving<br />

strategies are of particular interest and their diagnosis is required by the tutors.<br />

Another important advantage of the Bayesian Network approach is the derived graph which offers an easy to<br />

understand representation of the variables showing the nature of the interaction. Study of the graph could lead to<br />

usability improvements, as shown in the first study, by providing the most suitable help items or the possible<br />

actions related to the interaction dialogue flow and status. From an educational perspective, automated problem<br />

solving strategy recognition in open problem solving environments could substantially improve the learning<br />

process and support the tutors, when dealing with large sets of student data. That is, the educator could rapidly<br />

verify the adoption of specific problem solving strategies. In addition to this, real time implementation of the<br />

proposed approach, in the case of distance learning systems could aid diagnosing and evaluating the learning<br />

outcome and support teacher intervention in the form of adequate feedback.<br />

In general, it is argued that a more efficient and adaptive user system increases indirectly the pedagogical value<br />

of the environment because it contributes to the transparency of the tool A more intuitive flow of interaction<br />

enhances the learnability of complex tools such as those included in open problem solving environments. Our<br />

approach could be expanded to adapt the behaviour of the open educational environment in various aspects such<br />

as goal recognition, adaptive assessment regarding the expertise of the user, and adaptation of the interface (for<br />

example, adaptation of right click pull down menus, together with a constant set of commands). Further research<br />

is needed in order to investigate these areas, together with a longitudinal evaluation of the effects of the<br />

developed module on the way it affects student learning within such an environment.<br />

References<br />

Avouris, N., Tselios, N. & Tatakis E. C. (2000). Development and Evaluation of a Computer-based Laboratory<br />

Teaching Tool, Journal Computer Applications in Engineering Education, 9 (1), 8-19.<br />

Bunt, A. & Conati, C. (2003). Probabilistic Student Modelling to Improve Exploratory Behaviour. Journal of<br />

User Modeling and User-Adapted Interaction, 13 (3), 269-309.<br />

Collins, J., Greer, J. & Huang, S. (1996). Adaptive Assessment Using Granularity Hierarchies and Bayesian<br />

Nets. In Proceedings of Intelligent Tutoring Systems, 569-577.<br />

Conati, C., Gertner, A. S., VanLehn, K. & Druzdel, M. (1997). On-line Student Modeling for Coached Problem<br />

Solving Using Bayesian Networks. In Proceedings of the 7 th International Conference on User Modeling, 231-<br />

242.<br />

Conati, C., Gertner, A. & VanLehn, K. (2002). Using Bayesian Networks to Manage Uncertainty in Student<br />

Modeling. Journal of User Modeling and User-Adapted Interaction, 12 (4), 371-417.<br />

163


Cooper, J. & Herskovits, E. (1992). A Bayesian method for the induction of probabilistic networks from data.<br />

Machine Learning, 9 (4), 309-347.<br />

Daskalaki, S., Kopanas, I. & Avouris, N. (<strong>2006</strong>). Evaluation of Classifiers for an Uneven Class Distribution<br />

Problem, Applied Artificial Intelligence, 20, 1-37.<br />

Dimitracopoulou, A. & Komis, V. (2005). Design principles for the support of modelling and collaboration in a<br />

technology-based learning environment. International Journal of Continuing Engineering Education and<br />

Lifelong Learning, 15 (1/2), 30-55.<br />

Ergazaki, M., Komis, V. & Zogza, V. (2005). High-school Students’ Reasoning while Constructing Plant<br />

Growth Models in a Computer-Supported Educational Environment, International Journal of Science Education,<br />

27 (2), 909-933.<br />

Glymour, C. & Cooper, G. (eds.), (1999). Computation, Causation & Discovery. AAAI Press/The MIT Press.<br />

Gudzial, M. J. (1993). Deriving Software Usage Patterns from Log Files. Georgia Institute of Technology. GVU<br />

Center Technical Report. 93-41.<br />

Heckerman, D. (1996). A Tutorial on Learning Bayesian Networks. Technical Report MSR-TR-95-06, Microsoft<br />

Research.<br />

Hellman, R. (1989). User Support: Revealing Structure Instead of Surface, Behaviour and Information<br />

Technology, 8, (6), 417-435.<br />

Holmes, G., Donkin, A. & Witten, I. H. (1994). Weka: a machine learning workbench. In Proceedings of the 2 nd<br />

Australian and New Zealand Conference on Intelligent Information Systems, Brisbane, Australia, 357-361.<br />

Jameson, A. (1995). Numerical uncertainty management in User and Student Modeling: An Overview of<br />

Systems and Issues. In User Modeling and User-Adapted Interaction, 5 (3/4), 193-251.<br />

Jeffreys, H. (1939). Theory of Probability. Clarendon Press, Oxford.<br />

Kinshuk, P. A. & Russell, D. (2002). Intelligent and Adaptive Systems. In Adelsberger, H., Collins, B &<br />

Pawlowski, M. J. (Eds.) Handbook on Information Technologies for Education and Training, Germany:<br />

Springler-Verlag, 79-92.<br />

Komis, V., Dimitracopoulou, A., Politis, P. & Avouris, N., (2001). Expérimentations exploratoires sur<br />

l’utilisation d’un environnement informatique de modélisation par petits groupes d’élèves, Sciences et<br />

Techniques Educatives, 8, (1-2), 75-86.<br />

Komis, V., Avouris, N. & Fidas, C. (2002). Computer Supported collaborative concept mapping: Study of<br />

Interaction, Education and Information Technologies, 7 (2), 169-188.<br />

Kordaki, M. & Potari, D. (1998). A learning environment for the conservation of area and its measurement: a<br />

computer microworld. Computers and Education, 31, 405-422.<br />

Luger, G. F. & Stubblefield, W. A. (1998). Artificial Inteligence – Structures and Strategies for Complex<br />

Problem Solving, (3 rd Ed), Addison Wesley.<br />

Mayo, M. & Mitrovic, A. (2000). Optimizing ITS Behaviour with Bayesian Networks and Decision Theory. In<br />

International Journal of Artificial Intelligence in Education, 12, 124-153.<br />

Mitchell, T. (1997). Machine Learning, McGraw-Hill, New York.<br />

Nathan, M. J. (1998). Knowledge and Situational Feedback in a Learning Environment for Algebra Sotry<br />

Problem Solving. Interactive Learning Environments, 5 (1), 135-159.<br />

Niedermayer, D. (1998). An Introduction to Bayesian Networks and Their Contemporary Applications,<br />

University of Saskatchewan, Technical Report 184-3-54440 1998.<br />

164


Norman, D. A. (1986). Cognitive Engineering in User-Centred System Design, Laurence Erlbaum Associates,<br />

Hillsdale NJ, 1986.<br />

Pearl, J. (1988). Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. San Mateo,<br />

CA.<br />

Squires, D. & Preece, J. (1999). Predicting Quality in Educational Software: Evaluating for Learning, Usability<br />

and the Synergy between them, Interacting with Computers, 11, 1999, 467-483.<br />

Stephenson, T. A. (2000). An Introduction To Bayesian Network Theory And Usage , Institut Dalle Molle<br />

d'Intelligence Artificielle Perceptive, Technical Report IDIAP-RR 00-03, 2000.<br />

Tselios, N., Avouris, N. & Kordaki, M. (2002). Student Task Modeling in design and evaluation of open<br />

problem-solving environments, Journal of Education and Information Technologies, 7 (1), 19-42.<br />

Suthers, D., Connely, J., Lesgold, A., Paolucci, M., Toth, E. E., Toth, J. & Weiner, A. (2001). Representational<br />

and Advisory Guidance for Students Learning Scientific Inquiry. In Forbus, K. and Feltovich, P. (Eds.), Smart<br />

Machines in Education. The MIT Press. 2001, ISBN 0-262-56141-7<br />

Von Glasersfeld, E. (1987). E. Learning as a constructive activity. In Janvier, C. (Ed.) Problems of<br />

representation in teaching and learning of mathematics. London: Lawrence Erlbaum associates, 3-18.<br />

Vomlel, J. (2004). Bayesian networks in educational testing. International Journal of Uncertainty, Fuzziness and<br />

Knowledge-Based Systems, 12, 83-100.<br />

Xenos, M. (2004). Prediction and assessment of student behaviour in open and distance education in computers<br />

using Bayesian networks. Computers & Education, 43 (4), 345-359.<br />

Witten, I. & Frank, E.(2005). Data Mining: Practical Machine Learning Tools and Techniques, 2 nd Ed. Morgan<br />

Kaufmann Publishers, San Francisco, USA.<br />

Woolf, B. P., Beck, J., Eliot, C. & Stern, M. (2001). Growth and Maturity of Intelligent Tutoring Systems. In<br />

Forbus, K. & Feltovich, P. (Eds.), Smart Machines in Education. The MIT Press. ISBN 0-262-56141-7.<br />

165


Goldberg, A. K., & Riemer, F. J. (<strong>2006</strong>). All Aboard - Destination Unknown: A Sociological Discussion of Online<br />

Learning. Educational Technology & Society, 9 (4), 166-172.<br />

All Aboard - Destination Unknown: A Sociological Discussion of Online<br />

Learning<br />

Andra K. Goldberg<br />

Doctoral student, College of Education, Northern Arizona University, USA<br />

Tel: 928 692 3015<br />

akg27@dana.ucc.nau.edu<br />

Frances Julia Riemer<br />

College of Education, Northern Arizona University, PO Box 5774, Flagstaff AZ 86011-5774 USA<br />

Tel: 928 523 0352<br />

Frances.Riemer@nau.edu<br />

ABSTRACT<br />

This paper is an attempt to describe the emergence and growing popularity of online distance education<br />

over the past 30 years through changing sociological lenses. Examining the re-casting of the electronic<br />

classroom through the euphoria of techno-positivism, the power-embedded analysis of Critical Theory of<br />

Technology (CTT), and the critique of postmodernism, the paper addresses the implications suggested by<br />

each theoretical interpretation. Using the metaphor of a high-speed train, we encourage administrators,<br />

instructors and technicians to stop and reflect on the destination, rather than simply marvel at the speeds at<br />

which we are traveling and the engine that powers our ride.<br />

Keywords<br />

Online classes, Distance education, Techno-utopianism, Critical theory of technology, Postmodernism<br />

Introduction<br />

Imagine trying to board a train that is already moving along the tracks. You manage to get on, but the directions<br />

to your seat are written in a foreign language. You recognize the letters, but the words seem unintelligible. The<br />

conductor is faceless yet clearly frustrated with your inability to sit and interact with your unseen fellow riders.<br />

Worse yet, you must get to your destination on time or you will not be able to disembark.<br />

This scenario is neither a nightmare nor a horror movie plot; it is instead analogous to a first experience in an<br />

online class. Rest assured though that you are not alone in your discomfort. Both students and instructors<br />

struggle to negotiate online instruction, a still somewhat new mode of teaching and learning. Although assuming<br />

a far more removed stance, social theorists have also disparaged similar technological changes in modes of<br />

instruction. Their denunciations date back to Plato's condemnation of the privileging of writing over dialog in<br />

teaching – ironically while conducting his criticism in written form (Plato, 1961). Analyzing online education in<br />

terms of social theory is challenging because like Plato, theorists criticize technology but find themselves<br />

dependent on the very same tools, e-mail and computerized word-processing programs for example, that they<br />

criticize (Nobel, 1997).<br />

Mindful of this inherent paradox between message and medium, in this article we attempt to examine online<br />

education, the most recent entwining of technology and education. We begin with the birth of the Internet -<br />

arguably January 1, 1983 when TCP/IP (Transmission Control Protocol/Internet Protocol) was established as the<br />

standard protocol for Internet transmissions (Cutshall, 2003). Access to the Internet enabled computers of all<br />

types to interact with one another, resulting in the rapid growth of the World Wide Web and the beginning of a<br />

new type of distance education, online courses. Defined as “any formal educational process that occurs with the<br />

teacher and the student separated by either time or distance” (Davey, 1999, pp. 44-45), distance learning in an<br />

online format combines synchronous and asynchronous interaction. This combination is then packaged and<br />

delivered to anyone with a computer, Internet access, and the often hidden prerequisite of basic technical skills.<br />

Embraced as “the educational pedagogy of the future” (O’Malley, 1999, p. 3), distance learning, enhanced by<br />

web based technology, has made great strides since first offered in the form of correspondence courses, and later<br />

as cable television courses and via interactive television. By 2000-2001 for example, 56 percent of all<br />

postsecondary institutions offered some version of online courses, up from 34 percent in 1997. This growth is<br />

particularly significant in the public sector, where 90 percent of all public, two-year and 89 percent of all public,<br />

four-year institutions offer courses from a distance (Wirt, Choy, Rooney, Provasnik, & Sen, 2004). Recent data<br />

reveals that secondary students also take advantage of this growth in course offerings. In the 2002-2003 school<br />

year, thirty-six percent of all public schools had students enrolled in distance education courses, with students in<br />

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the<br />

copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies<br />

are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by<br />

others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior<br />

specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.<br />

166


fifty-nine percent of those schools enrolled in online courses. Of particular relevance to this paper, in forty-eight<br />

percent of those districts, post secondary institutions delivered the online courses. Online learning has become a<br />

run-away train, racing forward so quickly that it appears as a blur to the pedestrian standing by the tracks.<br />

Our examination of this new technological context of learning and teaching is sociological in nature. In an<br />

attempt to understand the high-speed bullet train of changes technology has brought to education and the<br />

strategies employed for coping with these changes, we focus on the portrayal of online learning in both public<br />

and academic discourse. Our discussion of these modes of thought pivot around three widely divergent<br />

theoretical perspectives: techno-utopianism, an incontestable belief in technology as progress regardless of any<br />

difficulties that may arise (Lears, 2000), Critical Theory of Technology (CTT), a critique that equates digitizing<br />

data to the “legitimiz[ation] of knowledge that fits into that structure” (Spitz, 2000, p. 9), and postmodernism, an<br />

emergent theoretical perspective that focuses on the coupling of immediacy and isolation in the contemporary<br />

world (Neal, 1999). By tracing the discourse that defined the emergence, growth, and use of online learning over<br />

the past 20 years, we hope to make visible the gap that Agalianos (1996, p. 2) noted when writing, “The social<br />

processes and the political decisions involved in the production and ‘consumption’ of IT in education are largely<br />

ignored."<br />

Hopeful Beginnings<br />

From the beginning of education’s courtship with technology, what has come to be known as techno-utopianism<br />

has flourished. Within this techno-utopian stance, technology in the classroom has long been proclaimed to<br />

“offer the greatest potential to right what’s wrong with our public schools” (California Educational Task Force,<br />

cited in Oppenheimer, 1997, p. 53). In schools and campuses across the US, computer-assisted learning has been<br />

deemed capable of great accomplishments that include both the transformation of teaching and the improvement<br />

of students' academic and vocational performance (DeCastell, Bryson, & Jenson (2003). Like the general public,<br />

teachers have come to see computers as essential to improving student achievement. When Michigan’s governor<br />

announced her intention to cut funding for a laptop project that provides computers to sixth grade students, for<br />

example, a middle school teacher insisted, “If the funding ends for this program, in three to four years, Bear<br />

Lake schools will take a 5-year leap backward in education” (Wendland, 2005, p. 12). Such strong convictions<br />

reflect an unquestioning acceptance of and growing dependence on technology that is manifested by teachers in<br />

a wide range of school settings.<br />

At its most basic, techno-utopianism is based on the belief that technology will revolutionize society. Actively<br />

promoted by Wired magazine since the periodical’s inception in 1993, techno-utopians view the Internet, the<br />

most recent example of digital technology, as a colorblind mechanism that fosters participation and involvement,<br />

while freeing individuals from governmental constraints. Techno-utopians promote online instruction as both an<br />

alternative to traditional class attendance and a mechanism for maximizing service to students at the lowest cost.<br />

District personnel cited the availability to offer “courses not otherwise available at the school, …meeting the<br />

needs of specific groups of students and offering Advanced Placement or college-level classes” as reasons for<br />

having distance education courses in the district (Setzer, & Lewis, 2005, p. 14). In a 1998 advertising campaign,<br />

community colleges, in the form of the Community College Distance Learning Network, were able to target a<br />

new potential student base, the large immigrant populations in Chicago and Los Angeles, because of the<br />

development of more than 500 courses delivered via the Internet (Chronicle of Higher Education, 1998).<br />

According to techno-utopian texts, students in online courses have been divided into seven discrete groups: 1)<br />

corporate learners pursuing education to maintain or upgrade work-related skills, 2) professional enhancement<br />

learners seeking to advance or shift careers, 3) degree-completion adult learners working to complete a degree at<br />

a later stage of their lives, 4) college experience learners preparing for life, 5) pre-college learners doing<br />

postsecondary work prior to the completion of high school, 6) remediation and test preparation learners<br />

preparing for an examination or enrollment in another program, and 7) recreational learners enrolled in classes<br />

for personal enjoyment (Oblinger, Barone, & Hawkins, 2001). Regardless of specific rationale and motivation,<br />

these learners are unable to attend traditional classes either because work or family situations restrict their<br />

participation or because they do not live in geographic proximity to a college campus. As a public relations<br />

article on Blackboard’s website asserts: “When course content and activities are provided online, students no<br />

longer need to worry about accessing course materials. Students can complete assignments during their most<br />

productive times. Busy students can choose to download readings or take practice exams whenever it is more<br />

convenient, in the evening after kids are put to bed or at 4am during a bout of insomnia” (Blackboard, 2000, p.<br />

3).<br />

167


In addition to convenience, proponents of online education described this new virtual context as both more fair<br />

and less biased than traditional classrooms. As the context for discussion in college classrooms shifted to chat<br />

rooms and other modes of electronic networks, techno-utopians reported that "groups of students at different<br />

schools, even in different countries, work together on collaborative projects, comparing the results of<br />

environment studies or cross-cultural surveys and thereby learning not only the subject at hand but also other<br />

skills in social relationships just the kind of learning that the early critics of teaching machines were afraid<br />

computers would stifle" (Starr, 1996, p. 55).<br />

Others highlighted the benefit of learners’ disembodiment in a virtual learning environment. “At once, you may<br />

say, we have eradicated the pathology of the classroom: learners will no longer feel their very presence has<br />

generated an inscription on their bodies by others. Fat, thin, shy, squeaky-voiced, slow, boisterous, late, sleepy,<br />

hairy–the whole Seven Dwarfs roll call–will be irrelevant in the new virtual learning environment” (Beckett<br />

1998, p. 6). Online distance learning has been described by Starr (1996, p. 56) as “race-neutral, location-neutral,<br />

age-neutral, income-neutral, disability-neutral and would be gender-neutral except for the clue of first names.”<br />

Starr continues, “Student participation in the discussion is greater than in any of our face-to-face classes. Some<br />

kinds of personal warmth appear to be more freely exchanged in the absence of bodies.”<br />

The hopeful optimism surrounding online education has been so pervasive that both pundits and academics alike<br />

have predicted its role in the demise of the residential university. MIT mathematician Seymour Papert (1984 as<br />

cited in Cuban, 1996) claimed that computers would make schools obsolete. In a similar vein, management<br />

pundit Peter Drucker (Lenzner & Johnson, 1997, pp. 7 - 8) predicted that:<br />

Thirty years from now the big university campuses will be relics. Universities won’t survive. It’s<br />

as large a change as when we first got the printed book …totally uncontrollable expenditures,<br />

without any visible improvement in either the content or the quality of education mean that the<br />

system is rapidly becoming untenable. Higher education is in deep crisis.…The college won’t<br />

survive as a residential institution.<br />

Conflicts Emerge<br />

But as the marriage of education and technology moved beyond the honeymoon stage, tensions began to emerge<br />

in the relationship. By the late 1990s, suspicions surfaced that the techno-utopianism that had been embraced by<br />

educators and the general public alike was in fact a result of “technopositivism,” a heavily marketed ideology<br />

that perpetuates a naïve faith in the promises of technology (Robertson, 2003b, p. 282). Critics claimed that<br />

teachers and schools were "easy targets" for assertive marketing campaigns that linked computer-assisted<br />

learning with issues of equity and access (Robertson, 2003a, p. 414). While techno-utopians proclaimed the cost<br />

effectiveness of electronic delivery, others saw universities struggling to upgrade instructional television<br />

systems, choosing between competing and costly course management software, and maintaining state of the art<br />

equipment in computer labs, libraries, and other points of access (Carmean & Haefner, 2003).<br />

The reality of online distance education also brought with it concerns about students’ actual access to<br />

technology, an increasingly apparent gap between requisite and actual skill levels, faculty members’ complaints<br />

about increased workloads, and a growing sense of isolation on the part of both instructors and students. While<br />

public discourse continued to portray the Internet as available to all, able to transcend class distinctions, and free<br />

of bias, research findings began to surface critiquing the expense of Internet access. According to a study<br />

conducted by Vanderbilt University’s Project 2000, access to the Internet was not randomly distributed, but<br />

correlated instead to income, education, and race (Novak & Hoffman, 1998). Conducted in January 1997 with a<br />

sample of nearly 6000 respondents, the study’s authors concluded that “household income explains home<br />

computer ownership; increasing levels of income correspond to an increased likelihood of owning a home<br />

computer, regardless of race” (Novak & Hoffman, 1998, p, 13). A report released the following year by the<br />

Benton Foundation (Goslee, 1998) attributed this gap to a lack of infrastructure, rather than to choices made by<br />

poor families. Exposing a pattern of telephone and cable companies’ prioritization of wealthier suburbs when<br />

wiring advanced systems, the report faulted the companies for neglecting to upgrade services in poor, inner-city<br />

neighborhoods. The report cited the mechanics of computer sales as problematic, claiming that a lack of<br />

financing options (i.e. credit cards, and checking accounts) severely limits the availability of Internet accounts to<br />

those living in poverty. In addition to infrastructure and hardware, access to online classes was also found to be<br />

dependent on the ability to afford a telephone, subscriber costs, and user fees.<br />

168


Instructors also found that students lacked the requisite technical skills to participate in an online class and did<br />

not always develop these skills as they progressed through the semester (Belcheir & Cucek, 2002; O'Brien &<br />

Renner, 2002; van Schaik, Barker, & Beckstrand, 2003). To quote Ronald Walker, professor of African<br />

American Studies at the University of Maryland, “It isn’t enough simply to have a personal computer – people<br />

need to integrate personal computers into their lives” (Floyd 1996, p. A20). Doubts began to surface concerning<br />

who actually benefited from online classes, students who would otherwise not have access to college classes, or<br />

already advantaged students for whom online learning added another layer of convenience. Distance education<br />

that required specialized software and Internet access may have been a boon for harassed professionals.<br />

However, it became increasingly apparent that for the unemployed student who needed these classes to get work,<br />

the requirements seemed to be yet another stumbling block (Gladieux & Swail, 1999).<br />

Blacker (1994, p. 17) labeled these tentative anti-technology sentiments a “critical theory of technology” (CTT).<br />

Driven by Marxists and neo-Marxists, critical theory asserts that rather than being neutral, technology<br />

advantages and profits the privileged (Feenberg, 1991; Marcuse, 1964). From a critical theory perspective, the<br />

much lauded cost-effectiveness of online learning has failed to address the additional burdens on faculty<br />

members, who struggle with the expanded time commitment required to convert a class to an online format and<br />

to attend to students who demand the immediate attention of faculty members to solve their technology-related<br />

problems. College administrators have addressed the increase in workload by providing faculty stipends and<br />

stressing the need for faculty buy-in (Natale, 2002; Serwatka, 2002; Thor, 1999). But according to critical<br />

theorists, administrators have given little to no consideration to the displacement of teacher in an online<br />

environment that has “a preference for substituting ‘delivery’ for ‘teaching’” (McWilliam & Taylor, 1998, p.29).<br />

Faculty members, in turn, have voiced skepticism about the de-peopling of electronic classrooms and the<br />

rendering redundant of their pedagogical skills. But in a climate in which market forces have already threatened<br />

their academic tenure, only a few have mounted any credible, cogent response to this pedagogical dislocation<br />

(Noble, 1998).<br />

Online Education in a Postmodern World<br />

The move from classroom-based to online education has also been viewed through a postmodern lens and<br />

positioned as an artifact of postmodern life. As Krishan Kumar (2001) observed: “The new society is now<br />

defined, and named, by its novel methods of acquiring, processing and distributing information…We do live in a<br />

world saturated with information and communication” (p. 97, 110). For post-modernists, wireless access has<br />

changed every aspect of our interaction. E-mail, instant messaging, online chat, and teleconferencing allow<br />

constant and immediate communication. At the same time, lives have also become more separate, and in turn,<br />

more isolated and private. Online classes reflect this dichotomy; students can access class at all hours of the day<br />

or night, yet they are increasingly isolated from their peers by an “electronic curtain” that constrains the<br />

development of a community of learning (Neal, 1999, p. 43).<br />

Discourse among students in online chat rooms and virtual conference centers mirrors this separateness. Neal<br />

(1999) describes student's online postings as “sequential position statements,” that simply refine their initial<br />

position rather than address those of their peers. According to Neal (1999), “It is too easy, in an electronic<br />

environment for students to escape the confrontations, challenges, and learning opportunities that are present in<br />

the classroom” (p. 43). While students may in fact choose this "self-imposed exile from communal conversation<br />

and action” when they register for online classes (Borgmann 1993, p. 2), university administrators have not fully<br />

grappled with the emotional distancing students and faculty experience while working within this disembodied<br />

context. In an unyielding campus climate of techno-utopianism, two mechanisms have become mainstays:<br />

counseling to steer students who voice a need for interaction away from online learning, and training to help both<br />

faculty develop online courses and students improve their technical skills. The presupposition here is two-fold.<br />

Students who favor distance learning “differ in their perceived learning needs from students who choose face to<br />

face formats” (Roblyer, 1999, p. 166). And that with support, anything can be taught and anyone can learn<br />

online.<br />

At the same time, students’ expectations of constant communication, instantaneous information, and immediate<br />

feedback have become painfully obvious to faculty members who teach in today’s instant culture. Instructors<br />

find themselves barraged by students who assume they are tethered to the computer, poised day and night to<br />

respond to e-mail requests or comment on discussion postings. Unlike scheduled office hours, online<br />

availability, 24 hours seven days a week has become the norm in our postmodern world. Yet within a postmodern<br />

frame, these changes are neither all negative nor all positive. While online classes have the potential to<br />

transform the teacher student relationship to one of service on demand, technology has also afforded faculty<br />

169


members increased flexibility. They no longer need to be physically present for office hours to answer student<br />

inquiries. As post-modernists proclaim, the collapsing of time and space, coupled with the disembodied nature of<br />

online instruction, has the potential to re-define the academy in ways we cannot yet imagine (Robertson, 2003b).<br />

Conclusion<br />

Drastically recast from an initial techno-utopianism to the power-centered analysis of CTT and the inherent<br />

contradictions of postmodernism, the cyber classroom is a manifestation of the pervasive changes engendered by<br />

the birth of the Internet. Managing and critiquing this pedagogical transformation is like painting a landscape<br />

from the window of le Train á Grande Vitesse (TGV). Feenberg (1999, p. 29) best describes the challenge when<br />

writing:<br />

Fortunately, how we design our new technologies is still an open question; the answer will<br />

decide which benefits and which limitations we end up with… If we can resist simplistic appeals<br />

to managerial efficiency and focus our efforts on sustaining the dialogue that has always been at<br />

the heart of the educational experience, then technology holds great promise; if not, then we face<br />

a great threat.<br />

Never immune to our own entrenchment in society, we recognize our techno-utopianism when we continue to<br />

hope that web-based technology holds great promise for learning and teaching.<br />

In conclusion, we reflect back to the train metaphor used in the title of this paper. When we doubt our own<br />

abilities to paint a landscape that has become a blur outside the window of our high-speed train, we contemplate<br />

the possibility of getting off at the next stop and re-evaluating the train itself. Unfortunately however, we fear<br />

that “full speed ahead” is almost always the dictum of the day. We encourage administrators, technicians, and<br />

faculty alike to make the occasional forced stop and reflect on where they are going and why. They need to ask<br />

the questions: What is the purpose and who profits? We hope they disembark, and take time to consider the<br />

destination, not simply marvel at the engine that is powering this high-speed train called Online Education.<br />

References<br />

Agalianos, A. (1996). Towards a sociology of education computing. Paper presented at the annual meeting of the<br />

American Educational Research Association. New York, NY.<br />

Beckett, D. (1998, April). Disembodied learning: How flexible delivery shoots higher education in the foot.<br />

Electronic Journal of Sociology, 3 (3), 5-10.<br />

Belcheir, M. J. & Cucek, M. (2002). Faculty perceptions of teaching distance education courses. Research<br />

Report 2002-02. Boise, Idaho: Boise State University.<br />

Blacker, D. (1994). Philosophy of technology and education: An invitation to inquiry. Philosophy of Education<br />

Society Yearbook, retrieved 30 May, <strong>2006</strong> from http://www.ed.uiuc.edu/EPS/PESyearbook/94_docs/BLACKER.HTM.<br />

Blackboard. (2000). Educational benefits of online learning: A Blackboard tip sheet, retrieved May 30, <strong>2006</strong><br />

from http://www.spokane.wsu.edu/net/help/content/BBdocs/Online_Learning_Benefits.pdf.<br />

Borgmann, A. (1993). Crossing the postmodern divide. Chicago: University of Chicago Press.<br />

Carmean, C. & Haefner, J. (2003). Next-generation course management systems. EDUCAUSE Quarterly, 1, 10-<br />

13.<br />

Chronicle of Higher Education (1998). Community colleges tout new distance-learning network. Edupage<br />

Newsletter. The American Educom Consortium. July 10, 1998.<br />

Cuban, L. (1996). Teachers and machines: The classroom use of technology since 1920. New York: Teachers<br />

College Press.<br />

170


Cutshall, S. (2003, March). Century schools, designing smarter, sleeker high-tech facilities. Techniques, 78 (3),<br />

19-21 and 60-61.<br />

Davey, K. (1999, Winter). Distance learning demystified. National Forum, 79 (1), 44-46.<br />

DeCastell, S., Bryson, M., & Jenson, J. (2003). Object lessons: Critical visions of educational technology,<br />

retrieved 30 May <strong>2006</strong> from http://educ.ubc.ca/faculty/bryson/gentech/objectlessons.htm.<br />

Feenberg, A. (1999). Whither educational technology? Peer Review, 1 (4), retrieved 30 May <strong>2006</strong> from<br />

http://www-rohan.sdsu.edu/faculty/feenberg/peer4.html.<br />

Feenberg, A. (1991). Critical theory of technology. New York and Oxford: Oxford University Press.<br />

Floyd, B. (1996). Program in Afro-American studies explores the racial gap in access to technology. The<br />

Chronicle of Higher Education, 43 (17), A19-A20.<br />

Gladieux, L., & Swail, W. (1999). The virtual university & educational opportunity: Issues of equity and access<br />

for the next generation. New York: The College Board Publications.<br />

Goslee, S. (1998). Losing ground bit by bit: Low income communities in the Information age. Washington DC:<br />

The Benton Foundation, retrieved May 30, <strong>2006</strong> from http://www.benton.org/publibrary/losingground/home.html.<br />

Kumar, K. (2001). Post-industrial to modern society: New theories of the contemporary world. In Halsey, A. H.,<br />

Lauder, H., Brown, P. & Wells, A. S. (Eds.) Education: Culture, economy, and society, New York: Oxford<br />

University Press, 96-112.<br />

Lears, J. (2000). Techno-Utopia? Tikkun, 15 (1), 39.<br />

Lenzner, R., & Johnson, S. (1997, March 10). Seeing things as they really are. Forbes, retrieved May 30, <strong>2006</strong><br />

from http://www.forbes.com/forbes/1997/0310/5905122a.html.<br />

Marcuse, H. (1964). One-dimensional man. Boston: Beacon Press.<br />

McWilliam, E., & Taylor, P. G. (1998, November). Teacher im/material: Challenging the new pedagogies of<br />

instructional design. Educational Researcher, 27 (8), 29-34.<br />

Natale, R. D. I. (2002). Ensuring quality from a distance. Community College Week, 14 (2), 4.<br />

Neal, E. (1999, Winter). Distance education: Prospects and problems. National Forum – Phi Kappa Phi Journal,<br />

79 (1), 40-43.<br />

Noble, D. F. (1997). Digital diploma mills: The automation of higher education. First Monday, 3 (1), retrieved<br />

May 30, <strong>2006</strong> from http://www.firstmonday.org/issues/issue3_1/noble/.<br />

Novak, T., & Hoffman, D. (1998). Bridging the digital divide: The impact of race on computer access Internet<br />

use. Vanderbilt University Project 2000, retrieved May 30, <strong>2006</strong> from<br />

http://elab.vanderbilt.edu/research/papers/html/manuscripts/race/science.html.<br />

Oblinger, D. G., Barone, C. A., & Hawkins, B. L. (2001). Distributed education and its challenges: An overview.<br />

Washington DC: American Council on Education Center for Policy Analysis.<br />

O'Brien, B. S. & Renner, A. L. (2002, June). Online student retention: Can it be done? Paper presented at the<br />

ED-MEDIA 2002 World Conference on Educational Multimedia, Hypermedia & Telecommunications. Denver,<br />

Colorado.<br />

O’Malley, J. (1999). Students perceptions of distance learning, online learning and the traditional classroom.<br />

Online Journal of Distance Learning Administration. 2 (4), retrieved May 30, <strong>2006</strong> from<br />

http://www.westga.edu/~distance/omalley24.html.<br />

171


Oppenheimer, T. (1997, July). The computer delusion. Atlantic Monthly, 280 (1), 45-62.<br />

Plato (1961). Collected dialogues. New York: Pantheon Books.<br />

Robertson, H. J. (2003a, January). Recycled promises. Phi Delta Kappan, 84 (5), 414-415.<br />

Robertson, H. J. (2003b, September/<strong>October</strong>). Toward a theory of negativity: Teacher education and information<br />

and communications technology. Journal of Teacher Education, 54 (4), 280-296.<br />

Roblyer, M. D. (1999). Is choice important in distance learning? A study of student motives for taking internetbased<br />

courses at the high school and community college levels. Journal of Research on Computing in Education,<br />

32 (1), 157-172.<br />

Serwatka, J. A. (2002). Improving student performance in distance learning courses. THE Journal, 29 (9), 46-51.<br />

Setzer, J. C. & Lewis, L. (2005). Distance education courses for public elementary and secondary school<br />

students: 2002-03 (NCES 2005-010). U.S. Department of Education. Washington DC: National Center for<br />

Education Statistics.<br />

Spitz, R. (2000). Is the Internet for everyone? Art in search of a better connected society. Paper presented at the<br />

10th International Symposium on Electronic Art (ISEA). Rio de Janeiro, Brazil. August 12.<br />

Starr. P. (1996, July-August). Computing our way to educational reform. The American Prospect, 27, 50-60.<br />

Thor, L. M. (1999). Keeping up with the Joneses. Community College Week, 11 (22), 4.<br />

van Schaik, P., Barker, P., & Beckstrand, S. (2003). A comparison of on-campus and online course delivery<br />

methods in Southern Nevada. Innovations in Education and Teaching International, 40 (1), 5-15.<br />

Wendland, M. (2005). School laptop project at risk. Detroit Free Press, retrieved April 18, 2005 from<br />

http://www.freep.com/money/tech/mwendland18e_20050418.htm.<br />

Wirt, J., Choy, S. P., Rooney, P., Provasnik, S., Sen, A. & Tobin, R. (2004). The condition of education, 2004.<br />

National Center for Education Statistics. Washington, DC: US Department of Education.<br />

172


Hsu, P.-S., & Sharma, P. (<strong>2006</strong>). A Systemic Plan of Technology Integration. Educational Technology & Society, 9 (4), 173-<br />

184.<br />

A Systemic Plan of Technology Integration<br />

Pi-Sui Hsu<br />

Department of Educational Technology, Research and Assessment, Northern Illinois University, USA<br />

phsu@niu.edu<br />

Tel: +1 815 7536025<br />

Priya Sharma<br />

Instructional Systems, Pennsylvania State University, USA<br />

psharma@psu.edu<br />

Tel: +1 814 8654374<br />

ABSTRACT<br />

The purpose of this article is to suggest a research-based systemic plan for educational researchers,<br />

practitioners, and policymakers involved in the change process to implement successful technology<br />

integration in the context of teacher education. This article provides a background about reform efforts in<br />

science education in the United States in recent years. This article also addresses technology tools that are<br />

responsive to reform efforts in the elementary science education and indicates the challenges of modifying<br />

elementary preservice teacher education to meet technology reform needs. The technology integration<br />

change process takes time and efforts; not all change can be sustained. Thus, a systemic plan that considers<br />

three major components, including people, process activities, and systems, to ensure successful technology<br />

integration is suggested.<br />

Keywords<br />

Technology integration, Science education, Preservice teacher education, Systemic inquiry<br />

Introduction<br />

This article is to suggest a research-based systemic plan for educational researchers, practitioners, and<br />

policymakers involved in the change process to implement successful technology integration in their contexts.<br />

We begin by describing the background of reform in science education in the United States, especially the<br />

changes that led to the emphasis on teaching science as inquiry. Next, a number of major types of technology<br />

tools that have the potential to support reform efforts are addressed. However, these technology tools are still<br />

rarely seen at school nowadays, which reflects a lack of technology integration in elementary science teacher<br />

education programs that play an important role of shaping school practice. The challenges of preparing<br />

elementary preservice teachers to teach science with technology are described and we end by proposing a<br />

systemic plan for technology integration in elementary science teacher education programs.<br />

Reform in science education<br />

In recent years, the United States has called for reform on science education in response to students’ poor<br />

performance in science tests and a general lack of interest in science. A number of professional scientific<br />

societies and organizations have taken the initiative in reforming science education. The National Research<br />

Council with the assistance of the National Academy of Sciences developed the National Science Education<br />

Standards (National Research Council, 1996), which addressed standards for teaching science and science<br />

education programs. The goals addressed standards for teaching science and science education programs. Based<br />

on the standards, it was recommended that science teaching take an inquiry-based approach and be adapted to<br />

meet the interests, abilities, and experiences of students. In this case, inquiry-based referred to “approaches and<br />

strategies for teaching and learning that enable learners to master scientific concepts as a result of carrying out<br />

scientific investigation” (National Research Council, 2001a, p. 187). Science teachers were encouraged to create<br />

an environment to enhance science understanding by encouraging students to communicate ideas with a<br />

community of learners. In addition, the standards specified the roles of colleges and universities in preparing<br />

teachers for implementing curricula that were consistent with the content standards.<br />

In 2000 and 2001, the National Research Council provided detailed guidelines for implementing inquiry-based<br />

science and classroom assessment (National Research Council, 2000, 2001b). The council suggested that in<br />

inquiry-based science, teachers serve as facilitators who set up a number of conditions for students to explore<br />

and investigate problems. Assessment becomes an integral part of science teaching and learning rather than a<br />

stand-alone activity.<br />

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the<br />

copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies<br />

are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by<br />

others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior<br />

specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.<br />

173


The basis of reform in science education<br />

With the increasing awareness of the importance of teaching quality, the National Academies established the<br />

Committee on Science and Mathematics Teacher Preparation (CSMTP) to identify critical issues in existing<br />

practices and policies for K-12 teacher preparation in science and mathematics in 1998. In 2001, Educating<br />

Teachers of Science, Mathematics, and Technology presented detailed recommendations for improving science<br />

teacher education from a systemic view based on extensive research and the committee’s careful examinations of<br />

practical experiences.<br />

From these reform documents, statements, and policies, four distinct characteristics can be identified as forming<br />

the basis of reformed science education:<br />

(1) Science teaching should employ an inquiry-based approach;<br />

(2) Assessment should attempt to ensure the quality of teaching and enhance the effectiveness of learning;<br />

(3) Science should be integrated with other disciplines and,<br />

(4) Applications of technology tools into science teaching and learning should be increased.<br />

The notion of science inquiry is the major focus of reform views of science teaching and learning (National<br />

Research Council, 1996, 2000). In the National Science Education Standards (National Research Council, 1996),<br />

two elements of scientific inquiry for science learners have been emphasized: abilities to do science inquiry and<br />

understandings about scientific inquiry. Some researchers proposed that science specific technology tools,<br />

including data collection tools, simulations and modeling tools, have the potential to facilitate learners’<br />

development from simple to sophisticated forms of scientific inquiry in recent years (Windschitl, 2000; Zembal-<br />

Saul, Munford, & Friedrichsen, 2002). However, the science specific technology applications in science teaching<br />

and learning are rarely seen or appropriately used in classrooms currently, particularly in elementary level. Most<br />

research reports are limited to discussing the integration of generic technology tools such as programming<br />

languages, spreadsheet, word processor, graphic programs, and the internet in elementary science teacher<br />

education programs (Cavanaugh, 2003; Linn, 2003; Skinner & Preece, 2003). For example, at Northern Florida<br />

University, preservice teachers were trained to use the internet as an instructional tool in an elementary science<br />

methods course. They learned search techniques and methods for evaluating web resources critically. They were<br />

then required to combine web resources with science content to create instructional units (Cavanaugh, 2003). In<br />

the UK, elementary teachers used a web site dedicated to elementary science to foster students’ understanding of<br />

science concepts and the use of concept-mapping software to stimulate discussion and assess students’<br />

understanding (Skinner & Preece, 2003). While these are important aspects of technology use and integration<br />

into science teaching, a more specific focus on the use of science-specific technological tools is missing and only<br />

a few teacher education programs have begun to integrate inquiry-based technology into science curriculum. For<br />

example, the University of Michigan has developed a science education program that prepares new teachers who<br />

understand physical science and are able to create meaningful learning experiences (Krajcik, Blumenfeld, Starr,<br />

Palincsar, Coopola, & Soloway, 1993). The Science Education Group at Penn State (Dana & Zembal-Saul, 2001;<br />

Zembal-Saul et al., 2002) also provided elementary preservice teachers with opportunities to teach science with<br />

technology and modeled the use of technology in lesson plans during the semester. These programs connect<br />

science with pedagogical courses, and provide an early teaching apprenticeship where students can develop<br />

lessons, discuss issues, and exchange ideas about using technology tools to enhance teaching. The next section<br />

proposes a number of technology tools that have the potential to support reform efforts.<br />

Technology tools for reforming inquiry-based science<br />

With emerging technological innovations, the use of technology tools in inquiry-based science learning has<br />

gained growing attention. Three types of inquiry-empowering technologies include data collection tools,<br />

simulations and modeling tools, and online collaborative tools (Windschitl, 2000). These three types of<br />

technologies assist elementary students in engaging in scientific phenomenon, conducting investigations, and<br />

communicating, and developing products (Zembal-Saul et al., 2002).<br />

Data collection tools<br />

Data collection tools encourage students’ active engagement with phenomena in a manner similar to scientists’<br />

engagement with scientific phenomena. The National Science Education Standards suggest that “a variety of<br />

technologies, such as hand tools, measuring instruments, and calculators, should be an integral component of<br />

scientific investigation. The use of computers for collection, analysis, and display of data is also part of this<br />

standard” (1996, p. 175). For example, in microcomputer-based laboratories (MBLs), students can observe<br />

174


graphs being produced by a microcomputer while an experiment is simultaneously being conducted. Nakhlen<br />

(1994) defined MBLs as “software which uses an electronic probe to collect information about a physical system<br />

and converts that information to a graphical system in approximately real time” (p. 368).<br />

In microcomputer-based laboratories, students are shown to develop science concepts, collect and analyze real<br />

time data, and visualize data much more quickly (Royuk & Brooks, 2003; Russel, Lucas, & McRobbie, 2003;<br />

Trumper & Gelbman, 2001). As mentioned earlier, the ability to graph and collect data is one of the integral<br />

practices of scientists in the course of their investigations. The microcomputer-based laboratories enable students<br />

to conduct difficult or previously impossible investigations (Bannasch, 2001). In addition, the microcomputerbased<br />

laboratories purport to strengthen students’ science graphing process and problem solving skills because of<br />

the capability of simultaneously collecting and graphing data. One example is probes, a microcomputer-based<br />

tool that can detect temperature, voltage, light intensity, sound, distance, dissolved oxygen and so on. Probes are<br />

now available for various hand-held devices, such as the Palm, through ImagiWorks, and the PASPORT System,<br />

through PASCO, thus increasing the possibility for dissemination and use in schools.<br />

Simulations tools<br />

Simulations combine video, pictures, computer graphics, text and interactivity to present students with<br />

phenomena that otherwise would be inaccessible, too hazardous, too time-consuming, or too expensive to<br />

observe. Using simulations, students can manipulate variables that would otherwise be too unethical, difficult, or<br />

impossible to do. For instance, Genscope (Horwitz & Christie, 2000) allows middle school students to breed<br />

various types of dragons to see the effect of selective breeding on observable characteristics, or phenotypes, in<br />

the offspring. Genscope allows students to ask “What if” questions and to do explorations that would otherwise<br />

not be feasible. Moreover, the linking between the visual characteristics of the offspring and the invisible world<br />

of chromosomes and DNA provides for unprecedented learning outcomes. Simulations allow students to ask<br />

“What if” questions, make predictions, and test out their ideas (Windschitl, 2000).<br />

Online collaborative tools<br />

Spitulnik, Stratford, Krajcik, & Soloway (1998) suggested that “teachers need to provide environments which<br />

support students’ inquiry, collaborating, and communicating” (p. 6). Such environments can help students to<br />

generate scientific understandings, which can be built up through email, threaded discussions, and chat rooms<br />

and other online communication tools.<br />

In recent years, several projects have made substantive efforts to create collaborative environments for students<br />

to get access to second-hand data and participate in discussions with people in the U.S and from around the<br />

world (Bombaugh, Sparrow, & Mal, 2003; Butler, &MacGregor, 2003; Howland & Becker, 2002). For example,<br />

the Global Learning and Observations to Benefit the Environment (GLOBE) Program (www.globe.gov),<br />

initiated by former Vice President Al Gore on Earth Day, April 22, 1994, is a hands-on international<br />

environmental science education program that establishes a partnership between students, their teachers, and the<br />

scientific research community. GLOBE provides opportunities for children to communicate and collaborate with<br />

scientists and other GLOBE students around the world. Students can participate in threaded discussions about a<br />

variety of topics and investigations with scientists. GLOBE also supports students in data collection and<br />

analysis. After data collection, students report their data through the Internet and compare their data to archived<br />

data collected in previous years. This project has been joined by 3,800 schools in the U.S. and has drawn schools<br />

from 50 countries to participate.<br />

In the preceding sections, this article discussed science education reform and the technology tools that have the<br />

potential to realize the goals of science education reform. In the next section we discuss the current challenges of<br />

preservice science teacher education in the area of technology integration.<br />

Challenges of elementary preservice science teacher education<br />

In this section, we discuss a number of problems in technology integration in teacher education programs that are<br />

equally evident for elementary science teacher education programs. A number of national reports (International<br />

Society for Technology in Education, 2002, 2005) have indicated that although teachers have more resources<br />

through technology, some of them have not received sufficient training in the effective use of technology. Most<br />

elementary preservice teachers in different content areas and disciplines know little about the effective use of<br />

175


technology and are not confident in using technology when they teach (Nonis & O’Bannon, 2002; US Congress,<br />

1995). They also lack an understanding of their role in implementing lessons with technology. Whetstone and<br />

Carr-Chellman (2001) conducted a survey, which revealed that elementary preservice teachers did not appear to<br />

see the importance of their roles in integrating technology in classrooms.<br />

In order to address these issues, research has shown that teacher education programs need to engage in<br />

substantive efforts to prepare preservice teachers for teaching with technology. Successful integration of<br />

technology in teacher education programs should provide preservice teachers with opportunities to use<br />

technology to advance content area learning (Bull, Willis, & Bell, 2000), and promote authentic learning<br />

experiences (Brush, Igoe, Brinkerhoff, Glazewski, Ku, & Smith, 2000).<br />

Specifically, colleges and universities should be aware of their ability to address elementary preservice teachers’<br />

competence in teaching with technology. Teacher education programs should consider all aspects of teacher<br />

education and have the ability to provide adequate opportunities for preservice teachers to apply what they have<br />

learned. Hargrave and Hsu (2000) presented results of a survey of instructional technology courses in preservice<br />

teacher education programs and concluded that more emphasis was placed on integrating instructional<br />

technology into the curriculum than on using technology for teacher productivity or personal use. However, most<br />

universities and colleges offered only one three-credit instructional technology course to prepare preservice<br />

teachers to teach with technology. In their survey study of investigating important factors in predicting<br />

preservice teachers’ professional uses of technology with K-12 students in the classroom, Dexter and Riedel<br />

(2003) found that setting high expectations for designing and delivering instruction using technology was<br />

effective in getting preservice teachers to use technology during clinical experiences. Yet, the results also<br />

indicated that preservice teachers wanted to have more access to technology and support and feedback from<br />

cooperating teachers and university faculty during field experiences.<br />

Systemic plan<br />

In light of challenges of elementary science education programs, this article suggests a research-based systemic<br />

plan to sustain the technology integration change process. The previous sections identified the reasons for<br />

elementary preservice teachers being ill-prepared to teach science with technology, including poor design of<br />

college-level courses and lack of integration of instructional technology courses into field experience. A<br />

systemic plan to address these shortcomings considers three major components: people, process activities, and<br />

systems to ensure successful technology integration.<br />

People<br />

1. Recruit key persons to form a leadership team.<br />

The first way to sustain technology integration is to form a leadership team by recruiting a minimal number of<br />

key faculty members or school teachers who possess expertise in elementary science education and share<br />

different experiences in technology integration. The faculty members fulfill certain types of functions, each<br />

focusing on design, administration, and establishing liaisons. Specifically, one leader focuses on the reconceptualization<br />

of the curriculum changes, monitors the progress in the implementation stage, and revises the<br />

curriculum either by informal feedback or conducting research. One leader acts like a facilitator to facilitate the<br />

changes by empowering other teachers to be receptive to changes within school by providing professional<br />

development. The other leader is responsible for outreach to the school community and securing necessary<br />

resources in the form of funding and human capital. It is suggested that these three major responsibilities be<br />

shared among the leadership team, consisting of faculty members, principals, school teachers, or school<br />

administrators.<br />

An example of leadership team is illustrated in the case of a team of university faculty in a science education<br />

program that sustained successful technology integration at the university level (Hsu, 2004). In this case, the key<br />

leadership team consisted of three faculty members, who had certain types of experience and adopted specific<br />

roles. The three faculty members fulfilled specific types of functions, each focusing on design, administration,<br />

and establishing liaisons. For example, one faculty member adopted the role of lead researcher in the context of<br />

technology integration because of her past research and teaching interest in technology integration in elementary<br />

science education. She, thus, conducted research with graduate students within the context of technology<br />

integration and based on the findings, she became responsible for refining course design and revising the<br />

176


technology integration process. The second faculty member assumed the role of coordinator with the university<br />

and the software companies because of the length of his tenure at the department and his knowledge of how and<br />

where to negotiate and secure resources. He, thus, ensured that the faculty members and graduate assistants<br />

participated in professional development workshops and obtained technological support from software<br />

companies. He also negotiated with administration to secure classrooms to set up technology tools and resources.<br />

The third faculty member, a former science teacher in a local school district, adopted the role of liaison between<br />

the science education program and various local school districts. She became responsible for providing adequate<br />

hardware, software and conceptual understanding of technology integration when these resources or support<br />

were unavailable in schools for integration into lessons by elementary preservice science teachers. Although<br />

these three leaders were responsible for different aspects, they held meetings and talked to each other to ensure<br />

that they were on the same track. This division of roles and responsibilities, in the light of individual abilities and<br />

interests, was a strong support for the technology integration process.<br />

A study of 13 Chicago elementary schools in an urban setting indicated another form of a school leadership team<br />

(Spillane, Diamond, Walker, Halverson, & Jita, 2001). The leadership team consisted of the principal, the<br />

science coordinator, and the assistant to the coordinator. The principal appointed a particular classroom teacher<br />

as a science coordinator. Collaborating with an assistant funded by the school, the science coordinator took<br />

advantage of the annual science fair and worked with other science teachers across all grade levels to integrate<br />

more inquiry-based science instruction and develop performance assessment rubrics to assess students’ projects.<br />

The principal doubled the weekly science instruction period, secured financial resources, established connections<br />

to external resources, including local universities, colleges, science institutions, and external science consultants,<br />

to empower change in science education. The leadership team showed a division among the functions of design,<br />

administration, and liaisons although some of the functions might overlap. Another example of a leadership team<br />

was identified in one elementary school in North Carolina (Franklin, 1993; Nesbit, DiBiase, Miller, & Wallace,<br />

2001).<br />

The joint effort of the leadership team members is a powerful force in sustaining change because leadership team<br />

members can deal with different aspects in the technology integration change process and thus contribute their<br />

expertise.<br />

2. Encourage the formation of a learning community.<br />

A learning community is essential in sharing ownership of the innovation and involving other members in a<br />

variety of activities. The formation of a learning community is important for several reasons: to generate a sense<br />

of shared ownership among program participants as well as to foster a lifelong learning experience for the<br />

stakeholders.<br />

One suggestion to form a learning community is to involve stakeholders in conducting action research. In Hsu’s<br />

(2004) study, the learning community consisted of a faculty member and four graduate students who were either<br />

course instructors or were interested in technology integration in elementary science. They conducted research<br />

within the context of technology integration in the elementary science methods course and field experience<br />

classrooms to examine practice critically. In the process, they developed a conceptual model for guiding<br />

technology integration in a science methods course. They also examined and evaluated the organization and<br />

content of the course assignments such as web-based portfolios to better assist elementary preservice science<br />

teachers in reflecting on their technology integration experience critically. Accordingly, the framework for<br />

guiding the web-based portfolio assignment underwent several revisions based on the group’s research for a<br />

period spanning six years from 1997 to 2003 (Hsu, 2004; Avraamidou & Zembal-Saul, 2002). The formation of<br />

the learning community helped the participants have shared ownership of the technology integration process,<br />

leading to a deeper commitment to the change process.<br />

The second activity for a learning community is to form study groups. Study groups read articles or books<br />

together, discuss the implications of ideas and engage in reflective dialogues, which lead to better instructional<br />

practice. For example, in addition to conducting research, the faculty member and the graduate students met<br />

weekly to discuss the readings assigned from the previous week (Hsu, 2004). During the meeting, they<br />

exchanged their ideas about obstacles encountered by their elementary preservice science teachers when they<br />

integrated technology into their field experience classrooms. Through these discussions, the study group<br />

participants were able to gain insight into specific problems and solve them by consulting each other in a short<br />

period of time. Reflective dialogues in the study group provided a basis for the changes of the design of<br />

curriculum in the following semester. The formation of the learning community sends a message to the<br />

177


participants that learning as a group can be much more powerful in sustaining the technology integration change<br />

process than learning individually.<br />

In the FIRST (Fund for the Improvement and Reform of Schools and Teaching) project initiated in North<br />

Carolina, teacher leaders and the other teachers used various activities to form a learning community to introduce<br />

inquiry-based science instruction (Nesbit et al., 2001). The teacher leaders co-taught lessons with other teachers<br />

in the school, which encouraged exchange of expertise between the teacher leaders and other teachers. The<br />

leaders also led other teachers to reflect on the challenges faced in the implementation of inquiry-based science<br />

instruction and solved the problems together. Teachers who showed resistance were invited to regular meetings<br />

to share their expertise; this involvement resulted in reduced resistance because of a growing understanding of<br />

the process and positive feedback from the teacher leaders. The teacher leaders also designed workshops where<br />

trainee teachers participated in role-playing to increase their confidence in delivering inquiry-based science<br />

lessons. In this case, the formation of the learning community creates a group learning environment, resulting in<br />

the reduced resistance from the stakeholders.<br />

At Ganado Primary School in Ganado, Arizona, the principal and teacher leaders built their learning community<br />

by involving teachers and staff in discussion about curriculum and instructional practices on a regular basis and<br />

by making time for collegial sharing (Cosner & Peterson, 2003). Other ways of building a learning community<br />

in schools include distributing weekly staff newsletters highlighting efforts of teachers who adopted innovative<br />

instructional strategies and communicating the expectations and vision with the teachers and staff.<br />

It is suggested that careful planning of these activities, including conducting action research, forming study<br />

groups, engaging in reflective dialogues, modeling teaching, collaborating on solving problems, and involving in<br />

regular discussions of instructional practices, bolsters the formation of a learning community in order to sustain<br />

technology integration.<br />

Process activities<br />

A variety of process activities are needed to ensure that the technology integration change process is able to<br />

sustain over time. Process activities include pilot-testing of technology tools and adopting web-based portfolios<br />

to scaffold teacher development.<br />

3. Pilot-testing the use of technology tools.<br />

Prior to any large-scale technology integration, it is important to pilot test the use of technology tools with a<br />

small group of end users and stakeholders within the authentic learning environment and collect as much<br />

feedback as possible from the users. It is also critical to allow some time to reflect on the feedback and to avoid<br />

rushing into the implementation of technology integration. Hsu’s (2004) research indicated that a key faculty<br />

member in the change process piloted different technology projects with a small group of elementary preservice<br />

teachers for two years before implementing the technology tools with the entire group of 180 elementary<br />

preservice teachers.<br />

Including stakeholders such as course instructors, students, and researchers (Hsu, 2004) in pilot testing can<br />

provide very important information. In the specific context of technology integration in an elementary science<br />

education program, for example, course instructors were able to describe their feelings about the technology<br />

tools and their appropriate integration into class and curriculum. Course instructors also provided assessment<br />

data that supported the utility of technology tools in enhancing learning; they could identify students’ technical<br />

and conceptual problems and suggest ways to improve student learning. Students could describe whether they<br />

learned better by using technology tools and whether the technology tools were easy to operate. Researchers<br />

could assess the project holistically in terms of technology integration and curriculum conceptualization, and<br />

suggest appropriate design refinements. Such testing provided empirical support regarding the efficacy of the<br />

proposed instruction in the elementary science methods course. Although pilot-testing appears to be a common<br />

procedure to help instructional designers make informed decision of the use of technology in organizations<br />

(White & Branch, 2001), it became an essential task to bring about the sustenance of the technology integration<br />

change process in this particular context.<br />

178


Pilot-testing appears to be a critical process activity in the technology integration change process. Although<br />

resistance might occur, it is imperative to take time to pilot-testing the technology tools with a small group of<br />

end users within the authentic learning environments.<br />

4. Using practice and reflective activities to scaffold teacher development.<br />

One important method for supporting pre-service teachers learning of technology-based science is the integration<br />

of reflection as part of the process. According to Lin, Hmelo, Kinzer, and Secules (1999), reflective thinking<br />

involves “actively monitoring, evaluating, and modifying one’s thinking and comparing it to both the expert’s<br />

models and peers” (p. 43). It is critical for preservice teachers to draw connection between the coursework,<br />

practice in clinical experience, and national standards. In addition, they need to adapt their knowledge and skills,<br />

cope the issues in their clinical experience classrooms and develop their repertoire of integrating technology to<br />

enhance teaching and learning. All of these skills require highly complex reflective thinking process. Therefore,<br />

it is essential to provide support and scaffolds to assist preservice teachers in developing reflective and critical<br />

thinking about these tasks in teacher education programs. While many options can support reflection, one<br />

popular form is a portfolio that integrates practice and reflection meaningfully.<br />

The notion of portfolio was proposed by the work of Lee Shulman (1988) and the creation of the National Board<br />

for Professional Teaching Standards in 1989. In teacher preparation programs, portfolio development has been<br />

demonstrated as being useful for preservice teachers in a variety of ways. First, portfolio development is a<br />

powerful tool to engage teachers in reflecting on their experiences, interrogating their practices, understanding<br />

their effects on students, and shaping their practices (Lyon, 1998; Schon, 1983). Second, portfolio development<br />

facilitates preservice teachers’ ability to connect theories and practices (Morris & Buckland, 2000).<br />

The portfolio has been represented in different media over time. Because of photocopying costs and storage<br />

problems, hypermedia portfolios gradually replaced paper-based portfolios. Web-based portfolio, as one type of<br />

hypermedia portfolio, is defined as “a user’s hypertextually linked set of electronic texts that have been created<br />

for and placed on the World Wide Web” (Watkins, 1996, p. 219). A number of studies have shown that<br />

hypermedia portfolios can promote students’ deep understandings of ideas in the development process<br />

(McKinney, 1998; Morris & Buckland, 2000). A similar conclusion was reached by Glasson and McKenzie<br />

(1999) in examining the development of multimedia portfolios for enhancing learning and assessment in an<br />

elementary science methods course. They concluded that “developing a hypermedia presentation enabled<br />

prospective teachers to construct and develop their ideas about teaching and learning. The portfolio documented<br />

the progress of preservice teachers as they developed curriculum and taught children at a local school and in the<br />

classroom” (p. 337).<br />

Land and Zembal-Saul (2002) investigated the influence of scaffolds in the Progress Portfolio, a generalized tool<br />

for articulation and reflection, on preservice teachers’ construction of scientific arguments within the context of<br />

an innovative science course that aimed at providing preservice teachers with experiences learning science using<br />

inquiry empowering technologies. The results indicated that the computer-based scaffolding supported<br />

articulation and reflection of evidence-based explanations. The preservice teachers showed increasing<br />

sophistication in their explanations, and the prompts within the Progress Portfolio seemed to stimulate preservice<br />

teachers to become more precise in their explanations, to offer justification, and to connect evidence with claims.<br />

They also suggested further research on varied scaffolding methods incorporated into computer-supported<br />

learning environments to facilitate reflective thinking.<br />

In the specific case of Hsu’s (2004) study, web-based portfolios were used to scaffold elementary preservice<br />

science teachers’ development in a science methods course. Two faculty leaders conducted a study about the<br />

progress of these teachers in developing web-based portfolios that included evidence and evidence-based<br />

justification to reflect on their experience in integrating technology into their science teaching within one<br />

semester in an elementary science methods course. The findings indicated that web-based portfolio development<br />

can support preservice teachers’ metacognition by making connections between evidence and justification<br />

explicit, and allowing them to express themselves creatively and save changes over time. Most importantly, the<br />

findings indicated that web-based portfolio development could engage students in meaningful reflection in<br />

technology integration in science teaching since these preservice teachers appeared to select more convincing<br />

evidence and provided stronger justification as the semester progressed.<br />

179


A hierarchy of Systems<br />

Identifying the function of different levels of the educational systems allows adoption of approaches that can<br />

strengthen synergy among different levels. In addition, foreknowledge of these levels allows one to address<br />

problems that may arise from these systems before and during the technology integration change process. In<br />

general, one can focus on the university level, the local community level, state level, and the broader public.<br />

5. Increase the involvement of the local community.<br />

At the local community level, it is essential to involve local school district teachers and students through the<br />

change process. Example activities include involving local school community in university-held events and<br />

delegating liaisons to regularly coordinate activities involving local community and local school district<br />

personnel. During the early stages of the technology integration change process at a northeastern university, Hsu<br />

(2004) found that in the first few years, barriers to technology integration included a lack of technology<br />

resources in school and lack of support from mentor teachers. Due to these deficiencies, elementary preservice<br />

teachers were unable to implement lessons with technology in their field experience classrooms. However, in the<br />

later stages of technology integration, Hsu (2004) indicated that inviting local elementary school students and<br />

teachers to visit the university appeared to be a successful intervention in sustaining the technology integration<br />

change process. During such visits, elementary preservice teachers practiced teaching lessons with technology<br />

with a small group of students and thereby gained opportunities to experience the effect of technology<br />

integration on science learning. These visits also provided in-service teachers with opportunities to observe how<br />

inquiry-based science lessons were taught using technology.<br />

White (2001) suggested the importance of reaching out to different interest groups in the community that may<br />

contribute to the enhancement in the areas of science and technology. These groups include the private sector,<br />

government, educational institutions, and the science and technology community. Taylor and Wochenske (2001)<br />

identified a number of strategies for a long-term business-school partnership that has enhanced science education<br />

reform efforts in the San Diego school districts for the past ten years. These strategies included holding annual<br />

science fair, inviting local scientists, university faculty and graduate students to schools, and engaging in-service<br />

science teachers in discussion with the science research community in after-school seminars.<br />

Professional development schools (PDS) are another method of establishing close relationships between<br />

university-based researchers and school-based educators. Grove, Strudler, and Odell (2004) suggested that it is<br />

critical to establish school district-university partnerships to assist student teachers in integrating technology in<br />

field experiences by implementing frequent professional development sessions. Mentor teachers helped novice<br />

teachers to build knowledge about how to teach in reform-minded ways with technology and how to mentor<br />

student teachers to teach in ways consistent with science reform standards. Mentor teachers were introduced to<br />

new practices and research in integrating technology with curriculum-based, student-centered activities by<br />

university faculty, which exposed them to new models for teaching and learning and learn to encourage novice<br />

teachers to teach in similar ways through modeling, practicing, and analyzing teaching together. Collaboration<br />

between school teachers and university faculty appears to be critical in sustaining the technology integration<br />

change process.<br />

6. Actively identifying funding opportunities and standards from the state department of education.<br />

At the state level, the department of education plays a critical role in the technology integration change process<br />

because it dictates curriculum design and funding of resources to sustain the changes. In the specific case of<br />

Hsu’s (2004) investigation of change in elementary science education, state standards greatly influenced the<br />

technology integration process. State academic standards in science informed the selection of topics for longterm<br />

projects every semester and the choice of technology tools. As an example, instructors designed units of<br />

lessons on watershed management to respond to the new standards of environmental ecology from the state<br />

department of education.<br />

In light of the economic hardship of state departments of education, a number of methods to secure external<br />

funding need to be identified. Cooper and Bull (1997) indicated that funding appeared to be an essential factor in<br />

technology integration because technology software and hardware need to be updated every year. They<br />

suggested that deans and other leaders, individually or collectively, should educate university presidents,<br />

provosts, state boards of education or professional practices boards and state legislatures on the importance of<br />

180


adequate technology resources. Bauer (2001) specified four major funding sources, including foundation<br />

funding, corporation grants, individual giving, and federal and state grants. Among these resources, federal and<br />

state grants place the greatest impact on technology resources in educational settings. Thus, it is critical to<br />

recognize the importance of funding in technology integration, to identify where the funds can be granted, and to<br />

identify means to obtain the funds to sustain technology integration.<br />

7. Identify means to collaborate with and gain recognition from national professional organizations and software<br />

vendors.<br />

At the national level, it is essential to identify means to gain recognition from accreditation agencies, to attend<br />

national contests or reviews and meetings of professional organizations, and to disseminate experience by<br />

authoring articles in journals, and to develop partnerships with educational software companies for appropriate<br />

support. Hsu (2004) identified that positive attention from an accreditation agency was critical in sustaining<br />

technology integrations. The faculty members and course instructors within the science education program<br />

gained confidence and momentum in technology integration because of recognition by National Council for<br />

Accreditation of Teacher Education (NCATE), an accreditation organization. Published journal articles and<br />

conference papers drew the attention and interest of people working in similar areas of research, which<br />

reinforced the accomplishments of faculty members’ and course instructors’ efforts in technology integration.<br />

Thus, in educational settings, rewards can accrue in a number of ways and it is not necessary to use money as a<br />

form of reward.<br />

Software companies can provide technical assistance and buy-in. They can also offer professional development<br />

workshops for the potential users and learners. Cooper and Bull (1997) suggested the establishment of<br />

partnerships between the educational programs and vendors of educational software to ensure that students<br />

receive appropriate exposure to software. They further suggested the National Council for Accreditation of<br />

Teacher Education (NCATE) and other organizations that establish technology standards for teacher education<br />

standards could take the initiative to implement such arrangements with educational vendors. In Hsu’s (2004)<br />

study, software companies were very supportive of the science education program and provided additional tools<br />

and professional training for the faculty members and course instructors. The companies also worked diligently<br />

to develop compatible technologies and more elementary-friendly tools based on faculty members’ and course<br />

instructors’ feedback. You (2001) provided a number of practical suggestions for developing school-business<br />

partnerships that promote educational reform in science and technology. These suggestions include aiming for a<br />

long-term relationship, getting top-level commitment, deciding on goals, looking for partnership opportunities<br />

that fit in school’s situation, being flexible, building on little success and publicizing your efforts, and assigning<br />

a staff person or outreach coordinator to develop partnerships.<br />

As discussed in the previous sections, the technology integration change process can involve different levels of<br />

systems. It is essential to identify the impacts that different systems might have and to exercise strategies to<br />

reinforce strengths of and tackle possible resistance from these systems.<br />

Conclusion<br />

Although we suggest a research-based systemic plan for integrating technology in elementary science education,<br />

a number of areas remain to be answered. First, a leadership team has proved as a powerful force in sustaining<br />

technology integration in elementary science education. However, it is important to further study methods for<br />

recruiting key persons with expertise in science education and keeping them involved in the technology<br />

integration change process. It is also important to identify and clarify mechanisms that allow for sharing<br />

responsibilities in a leadership team.<br />

Second, a learning community can be very powerful in sustaining the technology integration change process in<br />

elementary science education. However, we need to identify how to designate experts with vision to serve as<br />

catalysts for forming and leading the learning community. Additionally, research and practice must explore the<br />

design of a variety of significant, motivating, and relevant activities in which to engage stakeholders. Also, for<br />

busy teachers, it is important to identify methods to motivate them to engage regularly in aforementioned<br />

activities.<br />

Third, the technology integration change process cannot be sustained if support from different systems is<br />

insufficient. We identified a number of systems that have influence on technology integration. It is important for<br />

181


the change agents to recognize the resources and inhibitors within the systems and accordingly assign different<br />

weights while evaluating the success of the implementation plan in different contexts.<br />

This article provided the background of reform in science education in the United States in recent years. This<br />

article also addressed major types of technology tools that are responsive to reform efforts. In addition, this<br />

article indicated the challenges of elementary preservice teacher education and justified why the technology<br />

integration change process is worthy of further examination. Change takes time and efforts; not all change can be<br />

sustained. Thus, a systemic perspective was provided for educational researchers, practitioners, and<br />

policymakers involved in the change process to plan for successful technology integration in their contexts.<br />

References<br />

Avraamidou, L. & Zembal-Saul, C. (2002). Making the case for the use of web-based portfolios in support of<br />

learning to teach. The Journal of Interactive Online Learning, 1 (2), 1-19.<br />

Bannasch, S. (2001). Educational innovations in portable technologies. In Tinker, R. F. & Krajcik, J. S. (Eds.)<br />

Portable Technologies: Science Learning in Context, New York: Plenum, 121-166.<br />

Bauer, D. G. (2001). Finding the money. Principal Leadership, 1 (9), 24-28.<br />

Brush, T., Igoe, A., Brinkerhoff, A., Glazewski, K., Ku, H. Y. & Smith, T. C. (2000). Lessons from the field:<br />

Integrating technology into preservice teacher education. Journal of Computing in Teacher Education, 17 (4),<br />

16-20.<br />

Bombaugh, R., Sparrow, E. & Mal, T. (2003). Using GLOBE plant phenology protocols to meet the “National<br />

Science Education Standards.” American Biology Teachers, 65 (4), 279-285.<br />

Bull, G., Willis, J., & Bell, L. (2000). A user’s guide to the CITE journal. Contemporary Issues in Technology<br />

and Teacher Education, 1 (1), 4–8.<br />

Butler, D. M. & MacGregor, I. D. (2003). GLOBE: Science and education. Journal of Geoscience Education, 51<br />

(1), 9-20.<br />

Cavanaugh, C. (2003). Information age teacher education: Educational collaboration to prepare teacher for<br />

today’s students. Tech Trend, 47 (2), 24-28.<br />

Cooper, J. M., & Bull, G. (1997). Technology and teacher education: Past practice and recommended directions.<br />

Action in Teacher Education, 19, 97-106.<br />

Cosner, S, & Peterson, K. (2003). Building a learning community. Leadership, 32 (5), 12-15.<br />

Dana, T. M. & Zembal-Saul, C. (2001, March). Learning to teach with technology. Paper presented at the<br />

Annual Meeting of the Society for Information Technologies in Teacher Education (SITE), Orlando, FL.<br />

Dexter, S. & Riedel, E. (2003). Why improving preservice teacher educational technology preparation must go<br />

beyond the college’s walls. Journal of Teacher Education, 54 (4), 334-346.<br />

Franklin, M. E. (1993). Statewide Improvement in Elementary Mathematics and Science Education through Peer<br />

Teacher Training (Final Report of Project R 168D00258-92), Chapel Hill, NC: Mathematics and Science<br />

Education Network, University of North Carolina at Chapel Hill.<br />

Glasson, G. E. & McKenzie, W. L. (1999). The development of a multi-media portfolio for enhancing learning<br />

and assessment in a K-8 science method class. Journal of Science Teacher Education, 10 (4), 335-344.<br />

Grove, K., Strudler, N. & Odell, S. (2004). Mentoring toward technology use: Cooperating teacher practice in<br />

supporting student teachers. Journal of Research on Technology in Education, 37 (1), 85-109.<br />

Hargrave, C. P. & Hsu, Y. (2000). Survey of instructional technology courses for preservice teachers. Journal of<br />

Technology and Teacher Education, 8 (4), 303-314.<br />

182


Horwitz, P. & Christie, M. (2000). Computer-based manipulation for teaching scientific reasoning: An example.<br />

In Jacobson, M. & Kozma, R. B.(Eds.) Innovations in Science and Mathematics Education: Advanced Designs<br />

for Technologies and Learning, New York: Lawrence Erlbaum Associates.<br />

Howland, D. & Becker, M. L. (2002). GLOBE- The science behind launching an international environmental<br />

education program. Journal of Science Education and Technology, 11 (3), 199-210.<br />

Hsu, P. (2004). A Case Study of the Change Process of Integrating Technology into An Elementary Science<br />

Methods Course from 1997 to 2003, Unpublished doctoral dissertation, University Park, PA: The Pennsylvania<br />

State University.<br />

International Society for Technology in Education. (2002). NETS for teachers – Preparing teachers to use<br />

technology, Eugene, OR: ISTE.<br />

International Society for Technology in Education. (2005). The new national education technology plan,<br />

Eugene, OR: ISTE.<br />

Krajcik, J., Blumenfeld, P., Starr, M. L., Palincsar, A., Coppola, B. & Soloway, E. (1993). Integrating<br />

knowledge bases: An upper-elementary teacher preparation program emphasizing the teaching of science. In<br />

Rubba, P., Campbell, L. & Dana, T. (Eds.) Excellencing Educating Teachers of Science, Aubum, AL:<br />

Association for the Education of Teachers in Science.<br />

Land, S. M. & Zembal-Saul, C. (2003). Scaffolding reflection and articulation of scientific explanations in a<br />

data-rich, project-based learning environment: An investigation of progress portfolio. Educational Technology<br />

Research and Development, 51 (4), 65-84.<br />

Lin, X., Hmelo, C., Kinzer, C. K., & Secules, T. J. (1999). Designing technology to support reflection.<br />

Educational Technology Research and Development, 47 (3), 43-62.<br />

Linn, M. C. (2003). Technology and science education: starting points, research programs, and trends.<br />

International Journal of Science Education, 25 (6), 727-758.<br />

Lyon, N. (1998). Portfolios and their consequences: developing as a reflective practitioner. In Lyons, N. (Ed.)<br />

With Portfolio in Hand: Validating the New Teacher Professionalism, New York: Teachers College Press, 247-<br />

264.<br />

McKinney, M. (1998). Preservice teachers’ electronic portfolios: Integrating technology, self-assessment, and<br />

reflection. Teacher Education Quarterly, 85-103.<br />

Morris, J. & Buckland, H. (2000). Electronic portfolios for learning and assessment. Paper presented at the<br />

Society for Information Technology and Teacher Education conference, San Diego, CA.<br />

Nakhlen, M. B. (1994). A review of microcomputer-based labs: How have they affected science learning?<br />

Journal of Computers in Mathematics and Science Teaching, 13 (4), 368-381.<br />

National Research Council. (1996). National Science Education Standards, Washington, DC: National Academy<br />

Press.<br />

National Research Council. (2000). Inquiry and the National Science Education Standards, Washington, DC:<br />

National Academy Press.<br />

National Research Council. (2001a). Educating Teachers of Science, Mathematics, and Technology: New<br />

Practices for the New Millennium, Washington, DC: National Academy Press.<br />

National Research Council. (2001b). Classroom Assessment and the National Science Education Standards,<br />

Washington, DC: National Academy Press.<br />

Nesbit, C. R., DiBiase, W. J., Miller, A. S. & Wallace, J. D. (2001). In their own words: what science and<br />

mathematics teacher leaders say are important aspects of professional development, Chapel Hill, NC:<br />

Mathematics and Science Education Network, University of North Carolina at Chapel Hill.<br />

183


Nonis, A. S., & O’Bannon, B. (2002). A field-based initiative for integrating technology in the content areas:<br />

Using a team approach to preparing preservice teachers to use technology. Paper presented at the Society for<br />

Information Technology and Teacher Education, Nashville, TN.<br />

Royuk, B. & Brooks, D. W. (2003). Cookbook procedures in MBL physics exercises. Journal of Science<br />

Education and Technology, 12 (3), 317-324.<br />

Russel, D. W., Lucas, K. B. & McRobbie, C. (2003). The role of the microcomputer-based laboratory display in<br />

supporting the construction of new understandings in kinematics. Research in Science Education, 33 (2), 217-<br />

243.<br />

Schon, D. A. (1983). The Reflective Practitioner, New York: Basic Books.<br />

Shulman, L. S. (1988). A union of insufficiencies: strategies for teacher assessment in a period of education<br />

reform. Educational Leadership, 46, 36-41.<br />

Skinner, N. C. & Preece, P. F. (2003). The use of information and communications technology to support the<br />

teaching of science in primary schools. International Journal of Science Education, 25 (2), 205-219.<br />

Spillane, J. P. Diamond, J. B., Walker, L. J., Halverson, R. & Jita, L. (2001). Urban school leadership for<br />

elementary science instruction: Identifying and activating resources in an undervalued school subject. Journal of<br />

Research in Science Teaching, 38 (8), 918-940.<br />

Spitulnik, M. W., Stratford, S. J., Krajcik, J. & Soloway, E. (1998). In Fraser, B. J., & Tobin, K. G. (Eds.)<br />

International Handbook of Science Education, Great Britain: Kluwer, 363-381.<br />

Taylor, N. & Wochenske, J. (2001). The San Diego Science Alliance. In Thorson, A. (Ed.), Partnerships with<br />

Business and the Community, Columbus, OH: Eisenhower National Clearing House for Mathematics and<br />

Science Education, 37-38.<br />

Trumper, R. & Gelbman, M. (2001). A microcomputer-based contribution to scientific and technological<br />

literacy. Journal of Science Education and Technology, 10 (3), 213-221.<br />

US Congress. Office of Technology Assessment. (1995). Teachers and Technology: Making the Connections<br />

(GPO # 052-003-01409-2), Government Printing Office, Washington, DC.<br />

Watkins, S. (1996). World Wide Web authoring in the portfolio-assessed, (inter)networked composition course.<br />

Computers and Composition, 13 (2), 219-230.<br />

Whetstone, L. & Carr-Chellman, A. A. (2001). Preparing preservice teachers to use technology: Survey results.<br />

Tech Trends, 45 (4), 11-17.<br />

White, P. J. (2001). Finding Prospective Partners. In Thorson, A. (Ed.) Partnerships with Business and the<br />

Community, Columbus: OH: Eisenhower National Clearing House for Mathematics and Science Education, 27-<br />

29.<br />

White, B. S. & Branch, R. M. (2001). Systematic pilot testing as a step in the instructional design process of<br />

corporate training and development. Performance Improvement Quarterly, 14 (3), 75-94.<br />

Windschitl, M. (2000). Supporting the development of science inquiry skills with special classes of software.<br />

Educational Technology, Research and Development, 48 (2), 81-95.<br />

You, A. (2001). Guidelines for effective partnerships. In Thorson, A.(Ed.) Partnerships with Business and the<br />

Community, Columbus, OH: Eisenhower National Clearing House for Mathematics and Science Education, 21-<br />

23.<br />

Zembal-Saul, C., Munford, D. & Friedrichsen, P. (2002). Technology tools for supporting scientific inquiry: A<br />

preservice science education course. Paper presented at the Annual Meeting of the Association for the Education<br />

of Teachers of Science, Charlotte, NC.<br />

184


Cronjé, J., Adendorff, D. E., Meyer, S. M., & van Ryneveld, L. (<strong>2006</strong>). Surviving the shipwreck: what makes online students<br />

stay online and learn? Educational Technology & Society, 9 (4), 185-193.<br />

Surviving the shipwreck: what makes online students stay online and learn?<br />

Johannes C. Cronjé<br />

Faculty of Education, Groenkloof Campus, Universtity of Pretoria, Leyds Street, Groenkloof, South Africa<br />

jcronje@up.ac.za<br />

Tel: +27 82 558 5311<br />

Fax: +27 12 343 5065<br />

Debbie E. Adendorff<br />

Department of Curriculum Studies, Groenkloof Campus, University of Pretoria, 0002 South Africa<br />

i_adendorff@icon.co.za<br />

Tel: +27 825575295<br />

Salome M. Meyer<br />

Faculty of Medicine, University of Pretoria, 0002 Pretoria, South Africa<br />

salome.meyer@up.ac.za<br />

Tel: +27 828283380<br />

Linda van Ryneveld<br />

Deputy Director, Telematic Learning, Tshwane, University of Technology, Pretoria, 0001 South Africa<br />

vanryneveldl@tut.ac.za<br />

Tel: +27 828096379<br />

ABSTRACT<br />

Although much is written about reasons why students drop out of online courses, little is said about what<br />

makes them stay. This article reports on an experiment whereby online students were exposed to a learning<br />

experience modelled on the US television series Survivor. Twenty-four students were put into tribes and<br />

allowed to vote one another off the island at the end of each week. Students who were voted out of their<br />

tribes, were still on the course, but could no longer rely on the support of their peers. The course had a very<br />

high dropout rate, and students reported that the experience was extremely stressful. Yet there were fifteen<br />

students who completed the whole course. The question is why? This article identifies and discusses three<br />

aspects that contributed to the success of those who completed: The game metaphor, the roles and<br />

competencies of the facilitator, and the affective dimensions of peer support in a non-contact environment<br />

Keywords<br />

Online learning, Peer support, Games, Metaphor, Facilitator<br />

Introduction<br />

In his examination report of the first PhD that came out of this project Tom Reeves called it “a shipwreck of an<br />

online course”. Although it was an apt pun on the Survivor metaphor that we used, it was disconcerting to think<br />

of our project as a shipwreck. Yet, in reflection, we concluded that in a constructivist paradigm one has to learn<br />

from one’s mistakes; and in order to do so, one has to make them in the first place.<br />

In this article we reflect upon three PhD studies that investigated the Surfiver experiment. Twenty-four Masters’<br />

students enrolled for a six-week module on Internet-based distance education, which was to be presented almost<br />

entirely over the Internet. Instead of following a traditional approach whereby students would read and discuss<br />

documents and then finally write a term paper or an online test, the course was presented following the metaphor<br />

of a reality game show. Only fifteen students survived. The idea was to investigate the implications of using<br />

role-play and other games with adult online students. Games and case studies are not unknown in contact<br />

teaching of adults, particularly as icebreakers at training sessions, but there is little evidence of games being used<br />

in formal education, and no evidence of games being used in online education of adults.<br />

The question asked, was: “What are the conative implications of using games and a game-show metaphor to<br />

motivate adult students to complete an online learning course?” (Conative means: “to do with staying power”).<br />

We were specifically interested in what makes them stay, and not in what makes them drop out, as much<br />

research has already been done on barriers to online learning (Galusha, 1997). This phase of an ongoing<br />

development research project built on previous studies in the use of co-operative learning to improve online<br />

interaction (Cronje, 1997; Cronje & Clarke, 1999), as well as an investigation into the use of metaphor in<br />

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the<br />

copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies<br />

are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by<br />

others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior<br />

specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.<br />

185


maintaining the interest levels of online adult students (Cronje, 2001) and a small pilot study on affective aspects<br />

of online learning (Rauscher and Cronje, 2005). The main contribution of this article to the project is its enquiry<br />

on the implications of the game metaphor. Three classes of implications were identified, and each class formed<br />

the basis of an individual doctoral enquiry into the same case – the Surfiver online learning game. The first<br />

enquiry was thesis topic of the online facilitator, who investigated the implications of playing games with adult<br />

students in an online learning community (Van Ryneveld, 2005:85). Adendorff (2004) investigated the roles and<br />

competencies of an online facilitator, and the third study by Meyer (2005) considered how students coped with<br />

the lack of physical contact in an online learning environment.<br />

Course description<br />

The learning event that forms the basis of this article is six weeks of coursework in partial fulfilment of the twoyear<br />

Master’s Degree program in Computer-Integrated Education presented by the University of Pretoria. It is an<br />

optional course presented in the second year, but not as the final course, so that learners could enrol in another<br />

course should they deselect this one. By its nature as a distance course, it tends to have a high dropout rate, and<br />

students who drop out should have the opportunity to select another module before the end of the course. The<br />

course has been presented since 1995 and, in the spirit of development research, each iteration is enriched by<br />

what has been learnt from the previous one, while new facets are often introduced by way of an experiment. The<br />

complete curriculum can be found at http://hagar.up.ac.za/catts/ole/oro/outcomes.html.<br />

The students were highly motivated individuals who had already successfully completed more than two-thirds of<br />

the course. The students ranged from 23 to 55 years of age. There were eight male and sixteen female students.<br />

The course was presented in English; however, there were Afrikaans, Sotho, Xhosa, Zulu as well as English<br />

first-language speakers on the course. Within the first two weeks, nine students withdrew from the course while<br />

the remaining fifteen all completed successfully.<br />

Figure 1: The Surfiver island early in the course<br />

The course was supposed to be presented entirely online. No physical contact was allowed between students,<br />

except for an initial briefing session and a final de-briefing session. It was necessary, however, to have an<br />

emergency contact tribal council meeting early in the course to get floundering students on track.<br />

186


At the beginning of the course a Web-based island was created and students were required to erect their own<br />

shelters on the island, by way of modifying a graphic file in their home directories. Figure 1 shows the end result<br />

of their collaborative effort. Respecting the copyright of the television game, our game was called CyberSurfiver;<br />

a play on the original game show’s name and the fact that the participants surf the Internet. Five groups were<br />

formed; four competing tribes and one for those voted off the “island”.<br />

At the introductory meeting, the 24 self-selected participants were grouped into four groups according to their<br />

self-perceived Internet literacy. The rules of the game were explained to them while they sat in their groups to<br />

exchange email addresses, decide on a name for their group and to formulate a game plan. To stay true to the<br />

Survivor game format, they were given games to play. These were in the form of individual reward challenges<br />

and group immunity challenges. At the end of each week tribes had to vote one member off the team. The<br />

members who were voted off were not dropped from the course, but were put in a separate tribe where they<br />

would continue working, but they would no longer be eligible for the great prize, which in this case was not a<br />

million dollars, but a weekend holiday for one family. The winner would be the last one remaining.<br />

The tasks given to students included finding free online typing tutors, completing the tasks and sending in a<br />

screenshot of their results. Other tasks included building online jigsaw puzzles as well as uploading sound files<br />

containing their thoughts at a particular stage of the module. The final exam was a reflective essay on a topic<br />

relating to Internet-based learning in an online community. Topics included motivation, affective considerations,<br />

conflict management, etc.<br />

The academic importance of Cybersurfiver<br />

CyberSurfiver was an unusual experiment in Internet-based learning in higher education. Firstly, it is conducted<br />

almost entirely online and secondly, it uses a game as a means to achieve the course’s objectives. This course<br />

therefore had a wealth of information waiting to be extracted, analysed and, ultimately, published in order to<br />

positively contribute to the academic fraternity.<br />

The focus of this paper is to present the studies of three doctoral students in a manner which allows the reader to<br />

obtain a holistic understanding of online learning in general and, in this specific case, particular focus is given to<br />

the role that games play in adult online education. Firstly, the game metaphor is discussed. Thereafter, the roles<br />

and competencies of an online facilitator are investigated. This is followed by a discussion on affective<br />

considerations, as well as peer support, with regards to online learning. As this article covers the findings of<br />

three doctoral theses it will necessarily be rather superficial. The individual theses are available online and are<br />

listed in the references.<br />

Research methods<br />

The artefacts produced by the students, online discussions, individual emails to the facilitator and the final essays<br />

formed the main data sources of the investigations for the doctoral students. The transcripts of the primary<br />

briefing, the emergency tribal council meeting and the final council meeting also contributed to the research the<br />

doctoral students conducted. In addition, an independent professional facilitator conducted two focus group<br />

discussions and provided transcripts and analyses for triangulation. Two separate interview schedules (Creswell,<br />

1998:124) were designed. An independent interviewer and independent moderator were used to conduct semistructured<br />

interviewing to avoid any bias during the interview sessions. The transcripts of the interviews were<br />

used to corroborate evidence gathered during the focus groups. Verification methods for this study included:<br />

Member checking, peer reviews and the crystallisation of various points of view. The three doctoral students<br />

analysed the same basic data using different techniques to arrive at answers concerning the game metaphor,<br />

facilitator roles and competencies, and affective considerations and peer support. Each doctoral student also<br />

conducted an independent literature survey.<br />

The game metaphor<br />

Introducing games in adult learning can focus the adult’s attention. It can also garner positive attitudes,<br />

contribute to the adult’s motivation and improve their concentration, all while they actually enjoy the activity<br />

(Krasnor & Pepler 1980; Malone & Lepper 1987; Cheng & Van de Ven 1998; Cordova 1993; Garris Ahlers &<br />

Driscell, 2002).<br />

187


Focus group interviews with our students, as well as comments in their individual essays, showed that they were<br />

motivated by the immunity challenges and reward challenges. An analysis of the artefacts produced by students<br />

showed that even those who said that they did not like playing games still produced work of exceptional quality.<br />

The interviews revealed that they were more motivated by the small, intangible rewards (in the form of digital<br />

photos of prizes) that they received for the reward challenges than by the idea of being the overall winner of the<br />

weekend away.<br />

On the negative side, students felt that playing games was not central to their learning need. They perceived a<br />

mismatch between attaining the course objectives and reaching the game objectives; although in most cases these<br />

objectives were deliberately aligned with each other. A course objective, for instance, could be for students to<br />

improve their keyboard skills and the game objective was for students to see who types the fastest. The winner of<br />

this particular challenge, however, later admitted that she had asked her daughter to type for her, and sent in<br />

those results.<br />

Like Burke (2002), we also found that social interaction played a central role in stimulating students to<br />

participate in and, ultimately, complete the course. They used Yahoo messenger, email and (even in breach of the<br />

rules) the telephone to obtain help if they were stuck, or to share their joy in getting something right. The reward<br />

challenges intensified the course to such an extent that some students reported a feeling of “flow”<br />

(Csikszentmihalyi, 1990). This phenomenon is characterised by time going by unnoticed; ironically presenting a<br />

problem to some students because local telephone calls (to Internet service providers) are billed per second and<br />

as a result, excessive telephone charges are incurred.<br />

Students’ responses to the challenges of the games corresponded to O’Houle’s (1998) three categories of<br />

motivational styles: Goal, activity and learning orientation. An analysis of the students’ emails showed that goaloriented<br />

students disliked the games, as they were mainly interested in achieving high marks. Activity-oriented<br />

students enjoyed getting to grips with new technology and reported enjoying activities such as participating in an<br />

online demonstration of Interwise software building, online jig-saw puzzles or participating in a virtual photo<br />

shoot. The creativity of the metaphor intrigued the learning-orientated participants who enjoyed the unusual<br />

learning experience.<br />

Malone and Lepper (1987) identify challenge, curiosity, control, and fantasy as requirements for intrinsically<br />

motivating instruction. We found that in CyberSurfiver, challenge was a powerful motivating factor. Participants<br />

often reported working late into the night to complete difficult and challenging tasks.<br />

Participants received the topics of their assignments once a week. One student commented that she particularly<br />

appreciated that all topics were not issued at the beginning of the module because it had raised her levels of<br />

curiosity. The creative approach of the module ensured that students were keen to see what the next week’s<br />

assignments entailed − especially since they had just survived yet another turbulent week.<br />

Participants often indicated their despair at not being able to control their learning environment. They could not<br />

always control their computers or their connectivity, and battled to work collaboratively with other students<br />

whose learning styles, timetables and personalities differed from their own.<br />

The games in CyberSurfiver stimulated students to conceptualise artificially created situations. They were on a<br />

fantasy island. By engaging in the tribal and individual assignments and the reward and immunity challenges<br />

students had to use their imaginations to meet challenges, exercise control and experience interpersonal<br />

motivations.<br />

Even though the game metaphor of the course did stimulate and motivate the students to learn, they were still in<br />

need of guidance and support from an authoritative figure. This was made evident by the study done on the roles<br />

and competencies of an online facilitator.<br />

The roles and competencies of an online facilitator<br />

There is a misconception that teaching online is the same as teaching in the classroom (Broadbent & Legassie,<br />

2002; Zorfass, Remz & Ethier, 1998). Although individual authors never list more than seven online facilitator<br />

roles at a time, when synthesised, these roles amount to at least 23, but some overlap and could be integrated into<br />

already mentioned roles. Choden (2001) suggests that the various roles could be divided amongst several people,<br />

both in synchronous and asynchronous mode.<br />

188


To derive facilitator roles and competencies, White & Weight (2000) suggest various messages that online<br />

facilitators could write, while Anderson, Rourke, Garrison & Archer (2001:6-10) have a coding scheme for<br />

instructional design and organisation to facilitate discourse and direct instruction. Our own classification follows<br />

the Blignaut and Trollip (2003) taxonomy of facilitator messages, which distinguishes between those with<br />

academic and non-academic content. Academic content relates to the intellectual messages with sub-headings<br />

dealing with corrective, informative and Socratic messages. Non-academic content includes social,<br />

administrative, affective and other matters.<br />

Although there are several tasks for online facilitators (Full Circle Associates, 2002; Schuman, 2000), they still<br />

do not identify the online facilitator competencies. Moreover, competencies alone do not tell us the roles played<br />

by facilitators to make them “visible” in a web-based environment. Palloff & Pratt (1999) suggest that online<br />

facilitators have both intellectual and social roles.<br />

We attempted to isolate competencies that an online facilitator needs in the online environment. We selected the<br />

Work Profiling System (WPS) tool from Saville and Holdsworth Ltd (SHL) as the conceptual framework. The<br />

WPS has a cognitive component, (thinking competencies) an affective component, (people competencies) and a<br />

psychomotor component (energy competencies). Our analysis focused on the ‘visible’ roles played by the online<br />

facilitator and the identified competencies needed to fulfil in these roles. The data collected by the instruments<br />

already listed was analysed both by hand and using Atlas.ti. An unstructured face-to-face interview with the<br />

online facilitator was added.<br />

The analyses following the Blignaut & Trollip (2003) taxonomy indicated that the online facilitator provided<br />

comprehensive administrative information, supported the students in an affective manner and ensured that<br />

students received individual feedback on their postings. There was still no indication, however, which roles<br />

made the online facilitator “visible” to the students or how conflict is managed (White & Weight, 2000).<br />

Rourke, Anderson, Garrison & Archer (1999) suggest five units of analysis for research on computer<br />

conferencing: Proposition, sentence, paragraph, thematic and message units. We chose thematic units, defined as<br />

a “single thought unit or idea unit that conveys a single item of information extracted from a segment of<br />

information” (Rourke et al., 1999:60). It enabled us to capture the essence of each communication and more than<br />

one code could be assigned to each posting. The specific theme was “visibility”. Blignaut & Trollip’s (2003)<br />

categories were transformed into roles looking for indicators that would reflect “visibility” on the part of the<br />

online facilitator. We attempted to isolate what steps the students and the online facilitator recommended to<br />

improve the visibility of the online facilitator. Five roles of visibility (administrator, social supporter, instructor,<br />

guide and mediator) were identified.<br />

The administrator role conducts timely administration. The online facilitator divided students into groups,<br />

established communication channels, supplied important administrative information, posted reminders and<br />

provided support in terms of time management. The role of administrator involves course management, rather<br />

than dealing with academic issues. For instance, the facilitator discovered that students did not have one<br />

another’s email addresses. Students often complained about the high costs of online time and the amount of time<br />

the assignments took to complete. The facilitator accommodated the students where possible with deadlines for<br />

individual assignments, because she understood the constraints of work, family, health and other unforeseen<br />

events.<br />

The social supporter role ensures participation by, and retention of, students. This is achieved by providing<br />

emotional support and enhancing collaboration. Students need recognition, acknowledgement and positive<br />

feedback. The facilitator understood the students’ frustrations with the workload and the challenges they faced in<br />

terms of the technology and their fellow tribal members. She made a conscious effort to combat the feelings of<br />

isolation.<br />

The instructor should not take for granted that students all hear, read and internalise important information. The<br />

module started with a short contact session during which the nature of the Survivor metaphor was explained;<br />

students were divided into heterogeneous tribes; and given detailed instructions of where to find the<br />

CyberSurfiver: Introduction document as well as the first week’s assignments.<br />

The facilitator explained that students would experience “what it is like to be an online student”. Most of the<br />

students did not initially hear the information about the document. Others could not access the web-based<br />

version of Yahoo Groups, so they could not read the rationale of game and the outcomes of the module. The<br />

online facilitator, fulfilling the instructor role, undertook to assess all individual assignments halfway through the<br />

189


course and again at the end of the module. Apart from this official means of feedback the online facilitator<br />

stretched herself to provide the students with constant feedback online, both individually and for the group as a<br />

whole.<br />

The role of guide concerns building new knowledge. Just because students are not physically present does not<br />

mean that instructors can forget about their students. The facilitator guided them through the process by<br />

encouraging collaboration, keeping the students motivated, providing a challenge, putting them at ease and<br />

working one-on-one with individual students to fulfil their potential. The diversity of student skills was<br />

considered in the design of teaching strategies and curricula. Students with a strong Internet background used<br />

PHP to design their websites, while novices preferred easy-to-use web-hosting services.<br />

The mediator role ensures fair play. This module produced much conflict and some students missed out on the<br />

collaborative, group formation that the game should offer. Some could not co-operate, others would not and<br />

some never became involved, causing tremendous frustration for the rest. The facilitator managed the conflict<br />

within the groups and dysfunctional teams. Two students did not discuss their roles and contributions with each<br />

other, creating separate websites for one collaborative tribal assignment. Two students could not synchronise<br />

their online times, making all their collaboration asynchronous, very difficult and time consuming. Clashing<br />

individual personalities also affected the dynamics in the tribes.<br />

The five online facilitator roles where put through the Work Profiling System Job Analysis Questionnaires<br />

(JAQs) to determine people, thinking and energy competencies. The results indicated that the online facilitator<br />

needed a total of 13 competencies to be visible in the five roles of an online facilitator.<br />

The people competencies were: motivating and developing others, interpersonal sensitivity, teamwork and<br />

building and maintaining relationships. People competencies were mostly associated with the role of social<br />

supporter, where the facilitator established a friendly virtual environment. The thinking competencies were:<br />

judgment, information gathering, problem analysis, written communication skills, technical skills and<br />

competence. These were distributed fairly evenly across all roles, as were the energy competencies of selfconfidence,<br />

persuasiveness and oral communication skills.<br />

Once the roles and competencies of the online facilitator were examined, the question surrounding the emotional,<br />

or affective, responses of students who are being taught by a virtual lecturer were brought to the fore. What are<br />

the effects of being an “online student” and not being taught by a lecturer who is physically present? This<br />

problem was addressed; and the affective considerations of online learning as well as peer support were<br />

investigated.<br />

Affective considerations and peer support<br />

The data set generated by the methods already listed was analysed once again, but this time to determine how<br />

students cope with the absence of physical contact. The themes or meaning units were identified by means of in<br />

vivo coding (Graneheim & Lundman 2004:106; Holloway & Wheeler 2002:239,240). First-level coding was<br />

done by paraphrasing the words of the participants and collecting them into themes. Incorporating the themes<br />

into clusters (second level coding) and categories (third level coding) refined the themes (Holloway & Wheeler<br />

2002:239,240). Three main categories were identified: Curative factors, the process of development and<br />

inhibiting factors.<br />

Three clusters of themes form the first category, curative factors. These clusters were altruism versus<br />

individualism, communication, and internal drive or value system. Some themes indicated either altruistic or<br />

individualistic behaviour; resulting from feelings such as fear, trust, distrust, safety, insecurity, joy, and stress.<br />

Feelings of loneliness and isolation were experienced by a number of participants. Loneliness was a cause of<br />

increased levels of uncertainty and anxiety; particularly late at night and aggravated when they had a problem<br />

and requested assistance from their peers who would only read the email the next morning.<br />

Participants experienced both positive and negative emotions, but not ambivalence. In their digital<br />

communication they used emoticons to transfer their feelings, and compensate for the lack of personal contact.<br />

Nevertheless, they felt that e-mail was indirect and clinical and did not allow for spontaneous reactions. Sharing<br />

emotions (positive and negative) bound the participants together as a group. A feeling of closeness developed<br />

between tribe members, so that one of the participants felt it would be wrong to get rid of a person by voting<br />

her/him out, because they “came a long way together”.<br />

190


Feeling scared of being exposed also related to the self-image of participants. Some felt they had an image to<br />

uphold and should not be caught using English grammar incorrectly, as they were Master’s students. One<br />

particular participant indicated that she typed her e-mail message in MSWord, spell checked it, and then pasted it<br />

onto the e-mail composer.<br />

Another three clusters of themes were put together to form the second category of meaning, namely process of<br />

affective development. These phases included the initial phase: responding to requirements; the second phase:<br />

valuing, commitment and organising; and the third phase: internalisation. The phases correspond well with<br />

Krathwohl’s taxonomy of the affective domain (Huitt 2001).<br />

In the initial phase participants felt insecure, vulnerable and exposed because they did not know the level of their<br />

peers’ knowledge and skills:<br />

You sit there, totally naked and struggle. You do not know what the others knew [sic], and you<br />

know too little, but you do not know if they also know as little or less or more than you know.<br />

The intense module requirements initially made participants struggle alone. After feelings of chaos and distrust<br />

of their own abilities early in the module, they began to feel competent. During the second phase they took<br />

charge of their situation accepting their inabilities and abilities. They took active responsibility for their<br />

situations and motivated one another. During the third phase, participants made cognitive decisions regarding<br />

their efforts and those of their peers. Participants who completed the module saw themselves as people who do<br />

not give up, even though some considered doing that in the beginning.<br />

The third category of meaning was called inhibiting factors, and contained nine clusters: Giving up/being voted<br />

out; students’ lack of preparation; lack of technological support and knowledge, troublesome group selection,<br />

language, overload, high financial costs; and telephone service provider problems. This category addresses the<br />

intensity of the volition of the participants who stayed and completed the module regardless of experiencing<br />

many inhibitors. Participants felt that there was a definite gap in their knowledge and skill levels at the onset of<br />

the module. Participants experienced frustration due to technical problems and a lack of technical know-how,<br />

aggravated by the use of unfamiliar software.<br />

Conclusion<br />

In summary, we found that the highly competitive reality game show metaphor caused excessive stress to the<br />

students, to such an extent that one commented that “this was no game”. Nevertheless, those who survived<br />

reported a high level of satisfaction with what they had achieved, indicating, possibly some relationship between<br />

academic stress and a sense of achievement.<br />

The reasons for the successful completion were identified as the game metaphor, the skill of the facilitator in<br />

making herself “visible” over the Internet, and the emotional concern that the students showed for one another<br />

regardless of the absence of physical contact.<br />

The game metaphor allowed motivation through challenge, curiosity and fantasy. Students preferred intangible<br />

and insignificant email rewards, which seemed more attainable to them than the final prize, the weekend away.<br />

The downside of the metaphor was the perceived discrepancy between the goals of the games, and the learning<br />

goals. The main cause of conflict was the lack of availability of team members, lack of commitment and active<br />

participation, as well as contrasting personalities and strong individual wills.<br />

The online facilitator fulfilled five roles to attain full “visibility”. As administrator, she conducted timely course<br />

administration. She maintained social and emotional support. As an instructor she facilitated the learning<br />

process, but failed in providing explicit logistical guidelines. This was overcome by a database of frequently<br />

asked questions (FAQs). As a guide, she encouraged interactivity to foster the building of new knowledge. She<br />

did not risk losing students by abandoning them in cyberspace. As a mediator, she ensured fair play. If problems<br />

occurred, she intervened to resolve them. Filtering these competencies through a job profiling system produced<br />

five people competencies, five thinking competencies and three energy competencies.<br />

Three aspects were identified regarding affective considerations and peer support. These were curative factors,<br />

including altruism versus individualism, communication, and internal drive or value system. Three phases were<br />

identified in a process of affective development. These included an initial phase of responding to requirements; a<br />

191


second phase of valuing, commitment and organising; and a third phase of internalisation. Nine inhibiting factors<br />

were identified. These included the fear of dropping out, lack of preparation, lack of technical support and<br />

knowledge, group dynamics, work overload, financial concerns and connectivity problems.<br />

Students dropped out mainly because they could not work with the technology. Students stayed mainly because<br />

they enjoyed the game, felt looked after by the facilitator, and supported one another.<br />

More research should be conducted into the relationship between stress and a sense of achievement. Further<br />

work could include the development of a checklist for learners to determine how likely they would be to survive<br />

an online course, and an instrument to determine how likely a traditional facilitator would be to transfer to an<br />

online environment.<br />

Acknowledgements<br />

We acknowledge with thanks the excellent co-supervision of Prof. Dr Irma Eloff, and the pain of the<br />

participating students.<br />

References<br />

Adendorff, D. E., (2004). ‘An investigation into the roles and competencies of an online facilitator.’<br />

Unpublished Ph.D. thesis. University of Pretoria. Available at http://upetd.up.ac.za/thesis/available/etd-<br />

08232004-071632/<br />

Anderson, T. Rourke, L. Garrison, D. R. & Archer, W. (2001). Assessing Teaching Presence in a Computer<br />

Conferencing Context. Journal of Asynchronous Learning Networks, 5 (2), 1-17.<br />

Blignaut, A. S. & Trollip, S. R. (2003). Developing a taxonomy of faculty participation in asynchronous learning<br />

environments – an exploratory investigation. Computers & Education, 41 (2), 149-172.<br />

Broadbent, B. & Legassie, R. (2002). How to facilitate e-learning courses, retrieved June 26, <strong>2006</strong> from<br />

http://www.elearninghub.com/articles/how_to_facilitate_e-learning.html.<br />

Burke, K. (2000). Consumers turn to video and computer games for social interaction, retrieved June 26, <strong>2006</strong><br />

from http://www.games-advertising.com/assets/socialinteraction.htm.<br />

Cheng, Y. T. & Van de Ven, A. H. (1996). Learning the Innovation Journey: Order out of Chaos? Organization<br />

Science, 7, 593-614.<br />

Choden, A. (2001). How to Succeed as an Online Facilitator, retrieved June 26, <strong>2006</strong> from<br />

http://www.suite101.com/article.cfm/training_and_development/45384.<br />

Cordova, D. I. & Lepper, M. R. (1993). Intrinsic motivation and the process of learning: Beneficial Effects of<br />

Contextualisation, Personalisation, and Choice. Journal of Educational Psychology, 88 (4), 715-730.<br />

Cordova, D. I. (1993). ‘The effects of personalisation and choice on students’ intrinsic motivation and learning.’<br />

Ph.D dissertation, Stanford University.<br />

Creswell, J. W. (1998). Qualitative Inquiry and Research Design Choosing Among Five Traditions, Thousand<br />

Oaks, California: Sage Publications.<br />

Cronje, J. C. (1997). Interactive Internet: Using the Internet to facilitate cooperative distance learning. SA<br />

Journal of Higher Education, 11 (2), 149-156.<br />

Cronje, J. C. (2001). Metaphors and Models in Internet-Based Learning. Computers & Education, 37, 241-256.<br />

Cronje, J. C. & Clarke, P. A. (1999). Teaching 'Teaching on the Internet' on the Internet. SA Journal for Higher<br />

Education, 13 (1), 213-226.<br />

192


Csikszentmihalyi, M. (1990). Flow the psychology of optimal experience, New York: HarperPerennial.<br />

Full Circle Associates. (2002). Facilitator Qualities and Skills, retrieved June 26, <strong>2006</strong> from<br />

http://www.fullcirc.com/community/facilitatorqualities.htm.<br />

Galusha, J. M. (1997). Barriers to Learning in Distance Education, University of Southern Mississipi. Retrived<br />

June 26, <strong>2006</strong> from http://www.infrastruction.com/articles.htm.<br />

Garris, R. Ahlers, R. & Driskell, J. E. (2002). Games, motivation and learning: a research and practice model.<br />

Simulation & Gaming, 33 (4), 441-467.<br />

Graneheim, U. H. & Lundman, B. (2004). Qualitative content analysis in nursing research: Concepts,<br />

procedures, and measures to achieve trustworthiness. Nurse Education Today, 24, 105-112.<br />

Holloway, I. & Wheeler, S. (2002). Qualitative research in nursing, (2nd ed.) Oxford, UK: Blackwell.<br />

Huitt, W. (2001). Krathwohl et al.’s taxonomy of the affective domain. Educational Psychology Interactive:<br />

Taxonomy of the Affective Domain, Valdosta, GA: Valdosta State University, retrieved June 26, <strong>2006</strong> from<br />

http://chiron.valdosta.edu/whuitt/col/affsys/affdom.html.<br />

Krasnor, L. R. & Pepler, D. J. (1980). The study of children's play: Some future directions. In Rubin, K. H. (Ed.)<br />

New directions for child development: Children's play, San Francisco: Jossey-Bass, 1545-1558.<br />

Malone, T. W. & Lepper, M. R. (1987). Making learning fun: A taxonomy of intrinsic motivations for learning.<br />

In Snow, R. E. & Farr, M. J. (Eds.) Aptitude, learning, and instruction: Vol. 3. Cognitive and affective process<br />

analysis, Hillsdale, NJ: Erlbaum, 223-253.<br />

Meyer, S. M., (2005). ‘An investigation into the affective experiences of students in an online learning<br />

environment.’ Unpublished Ph.D. thesis. University of Pretoria.<br />

O’Houle, C. (1988). The inquiring mind, Madison, WI: University of Wisconsin Press.<br />

Palloff, R. M. & Pratt, K. (1999). Building Learning Communities in Cyberspace: Effective Strategies for the<br />

Online Classroom, San Francisco: Jossey-Bass Publishers.<br />

Rauscher, W. & Cronje. J. C.(2005). Online with Krathwohl: Affective aspects of learning in an online<br />

environment. South African Journal of Higher Education, 19 (3), 512-526.<br />

Rourke, L. Anderson, T. Garrison, D. & Archer, W. (1999). Assessing social presence in asynchronous, textbased<br />

computer conferencing. Journal of Distance Education, 14 (3), 51-70.<br />

Saville & Holdsworth Ltd (SHL). (1998). The Work Profiling System – Technical Manual. Printed in-house.<br />

Schuman, S. P. (2002). Facilitator Competencies from the Electronic Discussion on Group Facilitation,<br />

retrieved June 26, <strong>2006</strong> from http://www.albany.edu/cpr/gf/resources/FacilitatorCompetencies.html.<br />

Van Ryneveld, L., (2005). ‘Surviving the game: Interaction in an adult online learning community.’<br />

Unpublished Ph.D. Thesis. University of Pretoria. Available at http://upetd.up.ac.za/thesis/available/etd-<br />

03082005-220804/.<br />

White, K. W. & Weight, B. H. (Eds). (2000). The Online Teaching Guide – A Handbook of Attitudes, Strategies,<br />

and Techniques for the Virtual Classroom, Boston: Allyn & Bacon.<br />

Zorfass, J., Remz, A. & Ethier, D. (1998). Illustrating the potential of an online workshop through a case study<br />

example. Computer-Mediated Communication Magazine, 5 (2) retrieved June 26, <strong>2006</strong> from<br />

http://www.december.com/cmc/mag/1998/feb/zorfass.html.<br />

193


Kinzie, M. B., Whitaker, S. D., Neesen, K., Kelley, M., Matera, M., & Pianta, R. C. (<strong>2006</strong>). Innovative Web-based<br />

Professional Development for Teachers of At-Risk Preschool Children. Educational Technology & Society, 9 (4), 194-204.<br />

Innovative Web-based Professional Development for Teachers of At-Risk<br />

Preschool Children<br />

Mable B. Kinzie, Stephen D. Whitaker, Kathy Neesen, Michael Kelley, Michael Matera<br />

and Robert C. Pianta<br />

Curry School of Education, University of Virginia, Charlottesville, VA 22904-4265 USA<br />

Tel: +1 434-924-0835<br />

Fax: +1 434-924-1384<br />

kinzie@virgina.edu<br />

whitaker@virginia.edu<br />

kn3t@virginia.edu<br />

michaelkelley@virginia.edu<br />

mam4m@virginia.edu<br />

pianta@virginia.edu<br />

ABSTRACT<br />

Research suggests the importance of professional development for teachers of at-risk children, and while<br />

educational technologies can be harnessed to help support teaching practice in innovative ways, such<br />

programs are often limited in scale. My Teaching Partner (MTP) was designed to deliver a targeted<br />

program of professional development for teaching quality, integrated with early childhood curricula, to<br />

large numbers of teachers. It has been used by 235 teachers of at-risk preschool children across the<br />

commonwealth of Virginia. MTP links curricula with layers of web-based support, including an on-line<br />

consultant--a teaching expert who regularly observes, de-briefs, and extends teachers’ educational practice.<br />

The research-based curricula support the needs of at-risk Pre-K students in language, literacy, and social<br />

relationships; numerous video clips demonstrate effective teaching practice. In this paper, we describe the<br />

user-centered design process used and the support services and curricula delivered. The dynamic nature of<br />

the website is emphasized, including development practices which enable non-developers to easily<br />

contribute and maintain website contents. We discuss the strengths and weaknesses of this approach, and<br />

describe the field trial now underway, which includes examination of child outcomes, teaching quality, and<br />

teacher reflectivity. Evaluation data from the project are also summarized.<br />

Keywords<br />

Teacher professional development, Dynamic web design, Early childhood education, At-risk children<br />

Introduction & Background<br />

Needs of Early Childhood Teachers Working with At-Risk Students<br />

By the age of five, children growing up in poverty or related social disadvantage are often lacking in the<br />

language, literacy, and social relationship skills that are critical to school success (Pianta, 1999; Snow, Hemphill,<br />

& Barnes, 1991; Vernon-Feagans, 1996). Although increasing numbers of state-supported pre-kindergarten<br />

programs have been funded over the past decade, to meet the educational needs of these children (Blank,<br />

Shulman, & Ewen, 1999), many Pre-K teachers are in need of training in language and literacy development and<br />

ways to form positive relationships with children, foster emotional and social competence and self-regulation<br />

(Bowman, Donvan & Burns, 2001). In addition, large scale studies suggest overall mediocrity and high<br />

variability in classroom quality and practices, even when experienced, credentialed teachers use the same<br />

curriculum (Bryant, Clifford, Pianta, Howes, & Burchinal, 2002). As a result, a key factor in ensuring program<br />

quality, particularly at large scale, is the training and support of teachers in their implementation of<br />

scientifically-based classroom practices.<br />

Need for Innovation with Educational Technologies at Scale<br />

Innovative applications of technology to support teaching and learning are not uncommon. Further, evaluation<br />

outcomes for some of these initiatives have suggested their utility for supporting the learning and professional<br />

development of teachers. Few of these projects, however, are undertaken “at scale,” with the potential of<br />

addressing the needs of larger numbers of teachers and of making a significant impact on the practice of teaching<br />

and learning.<br />

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the<br />

copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies<br />

are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by<br />

others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior<br />

specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.<br />

194


Clifford and Maxwell (2002) have suggested that if a voluntary Pre-K program were offered to all four-year-olds<br />

across the United States, over 200,000 teachers would be required, significantly more than the 27,000 qualified<br />

teachers estimated as practicing in 2002. It is clear that if educational technologies are to help meet the training<br />

and professional development needs of these teachers, these efforts must be at scale.<br />

Innovative Technologies, At Scale, to Meet Teacher Needs<br />

To address the professional development needs of a large number of Pre-K teachers working with at-risk<br />

students, a team of five principal investigators at the University of Virginia’s Curry School of Education<br />

designed and developed MyTeachingPartner (MTP). MTP is an innovative, technology-assisted program that<br />

provides teachers with support from their own on-line consultant, a teaching expert who assists teachers by<br />

regularly (about twice each month) observing, de-briefing, and extending teachers’ educational practice. MTP<br />

also provides a set of web-based resources to support high quality teaching, including research-based educational<br />

curricula designed specifically to support the development of early language, literacy and social relationships in<br />

Pre-K students. The curricula are presented on the Web and illustrated with numerous video demonstrations of<br />

effective practice taken from actual Pre-K classrooms. At present, 235 teachers across the commonwealth of<br />

Virginia are participating in a two-year field trial of MTP, using our curricula every day in their classrooms.<br />

Among our outcome measures is a two-year follow-up on the effects of MTP on children’s language, literacy,<br />

and social development.<br />

In this paper we describe the user-centered design process which led to the development of MTP, including<br />

needs assessment and the design of on-line professional development support services and teaching curricula.<br />

We demonstrate our methods for dynamically generating all pages within this website, a development practice<br />

that enables non-developer content experts and administrative staff to quickly and easily contribute and update<br />

website content. Next, we describe the current research underway as part of our field trial, including examination<br />

of child outcomes (from baseline to two year follow-up for each child studied), effects on teaching quality, and<br />

development of teacher reflectivity as a function of consultation. Finally, we will discuss the completed and ongoing<br />

evaluations of MTP, and some of the implications.<br />

User-Centered Design Process<br />

Needs Assessment & Iterative Prototype Development & Testing<br />

From its inception, MTP has focused on the development of high quality Pre-K teaching practices and, through<br />

them, on developing children’s early language, literacy, and social/emotional competence. This approach is<br />

supported in naturalistic studies of child care and Pre-Kindergarten settings (Howe, et al., in press; NICHD<br />

ECCRN, 2002). The design of its methods and materials has evolved over time.<br />

Figure 1. Prototypic Website Flow Document<br />

195


Although this work is based on a strong foundation of prior research and theory building, the design process still<br />

began with a needs assessment. Ten graduate students spent seven weeks in an advanced instructional design<br />

course conducting this assessment. They began by consulting with teachers, educational experts, and specialists<br />

in emerging Internet-based communications technologies, among others. They went on to review models of<br />

consultancy and literature on Pre-K education and child development. Project goals were then articulated to<br />

describe outcomes for the Pre-K teachers, and a website prototype and initial design for the Internet-based<br />

Consultancy for teachers were mocked up. Although the prototypes were early ones, their form predicted many<br />

of the features of current MTP products. See Figure 1 for an example.<br />

Figure 2. Weekly Lesson Plan Options<br />

196


The prototypes, along with early drafts of the curriculum design, were evaluated by four Pre-K teachers in three<br />

rounds of prototype testing, and the prototypes were revised.<br />

Curriculum Design<br />

MTP instructional activities focus upon language and literacy as well as social and emotional competence. The<br />

MTP Language and Literacy Curriculum (Justice, Pullen, Hall, & Pianta, 2003) draws upon scientificallysupported<br />

practices for teaching phonological and written language awareness skills. The curriculum integrates a<br />

child-driven, highly-contextualized approach with a concentrated teacher-guided and explicitly targeted<br />

methodology for development of key language concepts. This is a model of best practice for early childhood<br />

language intervention programs (Justice & Kaderavek, 2004). There are six targeted skill areas within the MTP<br />

Language and Literacy Curriculum, which teachers address during 30 minute lessons, four times each week:<br />

Alphabet Knowledge, Narrative, Phonological Awareness, Pragmatics and Social Language, Print Concepts, and<br />

Vocabulary Linguistic Concepts. Teachers can either follow our sample weekly lesson plans or build their own<br />

weekly plans in response to the specific needs of children in their classrooms (see Figure 2).<br />

In its current field trial, MTP is also designed to support implementation of the PATHS (Promoting Alternative<br />

THinking Strategies; Greenberg, Kusche, Cook, & Quamma, 1995) curriculum, which focuses on development<br />

of healthy child-teacher relationships and classroom management. Teachers devote at least one 30 minute lesson<br />

to PATHS each week. Other methods for building teacher-child relationships are also employed, such as Banking<br />

Time (Pianta & Hamre, 2001). Thus, the MTP approach addresses major skill needs of Pre-K children, to<br />

encourage school readiness (Pianta, 1999).<br />

Figure 3. Video Demonstration<br />

197


Website Design<br />

The MTP website (www.myteachingpartner.net) offers a variety of supporting materials, including video<br />

demonstrations of the activities and of quality teaching practice, sample weekly lesson plans, professional<br />

development materials related to Quality Teaching (La Paro, Pianta, & Stuhlman, 2004) and Banking Time<br />

techniques. Examples of the demonstration videos and professional development resources are given in Figures 3<br />

and 4. As the website was being developed, we subjected it to two additional rounds of user testing with twelve<br />

pre-service teachers, with accompanying revisions. In its final form, the MTP website provides both a set of<br />

comprehensive instructional activities and resources to support high quality implementation of those activities.<br />

Consultancy Design<br />

Figure 4. Professional Development Resources for Quality Teaching<br />

The purpose of the MTP Consultancy is to provide teachers with a more intense, individualized form of support<br />

for high quality implementation--direct feedback on their teaching and opportunities to reflect on their practice.<br />

It is based on a collaborative professional development relationship between a teacher and an MTP consultant, in<br />

which systematic observation of classroom practice forms the basis of a partnership that works to support<br />

teachers’ high quality implementation of MTP activities and promote quality teacher-child interactions. The<br />

MTP consultancy process provides opportunities for weekly collaboration through a combination of technologies<br />

that enable regular communication. One form of communication occurs through videoconferencing (and/or<br />

telephone conversations, in the event of technical difficulties). Videoconferencing allows the consultant to see<br />

and hear the teacher in real time. The consultant and the teacher view video clips and share insights about the<br />

implementation of the activities. Another form of communication occurs through use of online journals and<br />

email. Figure 5 displays a consultant-teacher conference, and Figure 6 depicts the consultancy process.<br />

198


Dynamic Web Design<br />

Figure 5. Teacher-Consultant Video Conference<br />

Figure 6. The Consultancy Cycle<br />

A key to our success with a limited staff and budget was to create a dynamic web infrastructure for the entire<br />

project. Once we determined the functions that were needed, we crafted a framework for the project built upon a<br />

PHP/MySQL server. This has allowed contents to be created and entered by various staff members, and has<br />

enabled content to be updated quickly and easily at any time.<br />

Access Portals Created for Project Development by Non-Programmers<br />

We developed password-protected web interfaces for each type of MTP staff member, to assist them in<br />

contributing contents and managing their branch of operations. We describe some of these access portals in<br />

Table 1. Similar interfaces have been created for:<br />

Project Managers, to integrate all aspects of the project (e.g., when new teachers join the project, their data<br />

automatically populates all databases)<br />

199


Researchers, for review and coding videos from participating classrooms, and to review student assessment<br />

data<br />

Development/Technical Team, for production and review of demonstration videos (see Neesen, et al., 2005)<br />

and for review of web usage statistics.<br />

Benefits & Limitations of Dynamic Web Design<br />

Table 1. Access Portals for Project Staff<br />

Curriculum Developers use this interface<br />

for:<br />

Entering activity contents via a<br />

structured interface that prompts for<br />

each needed component (e.g., title,<br />

curricular area, materials, etc.)<br />

Formatting the activity display and<br />

linking to related materials<br />

Editing descriptions of existing<br />

materials<br />

Consultants have their own website through<br />

which, they can:<br />

Select a teacher to work with<br />

Identify and upload a video clip for that<br />

teacher’s current consulting cycle<br />

Provide prompts for teacher reflection<br />

Review teacher responses to those<br />

prompts<br />

Summarize and code their on-line chat<br />

discussion<br />

E-mail teachers any follow-ups<br />

The Administrative website allows project<br />

managers, consultants, and the<br />

development/technical team to:<br />

Access teacher information, including<br />

any technical support issues they<br />

have/had<br />

Submit a new technical issue for a<br />

given teacher<br />

Search the Frequently Asked Questions<br />

(FAQs) database to assist a teacher<br />

Suggest a new possible FAQ<br />

Get a quick calculation of the average<br />

time required to address technical<br />

issues<br />

A similar interface was developed to<br />

support the research functions of the project,<br />

including delivery of measures to teachers<br />

for data collection and systems for data<br />

export.<br />

As noted in Table 1, a dynamic approach to our interface has a variety of benefits:<br />

Quick updating of the contents of our teaching activities,<br />

Easy modification of the interface (only the template needs to be modified, not every page displaying a<br />

teaching activity),<br />

200


Flexibly linking of materials throughout the website, without reproducing materials or re-coding for their<br />

display (for instance, we use video clips extensively, both to demonstrate how to implement particular<br />

activities and, packaged differently with related commentary, as examples of Quality Teaching practice),<br />

Easier backup of site contents: each database is stored in just one file,<br />

“Ownership” and responsibility for site contents is shared across a wider array of individuals (MTP is made<br />

up of 11 full-time and 10 part-time faculty and staff, plus 6-8 graduate students at any point in time),<br />

lessening the work load for the development team and speeding the development and maintenance<br />

processes,<br />

Different types of permissions assigned to different individuals, allowing some to have administrative access<br />

to databases (e.g., the ability to create or delete records), and others to have “view only” access.<br />

However, dynamic design does have attendant limitations:<br />

Initial setup of website and database is time-consuming and required more sophisticated development<br />

expertise than static web page creation. We estimate spending six months total time of a systems developer<br />

to build all of the structures needed, in addition to the time normally required to design a web site.<br />

While editing the contents of the website and revising the layout are greatly simplified with a dynamic<br />

approach, it is less easy to add categories of information, as both the web page template and the database<br />

need to be revised.<br />

MTP Project Implementation & Outcomes<br />

Current Research Design<br />

In our current field trial, we are comparing the relative effects of different versions of the MTP professional<br />

development model. Participating teachers attended a one-day summer workshop on MTP, during which they<br />

received a laptop computer they have used to review MTP teaching materials. Teachers in each of the three<br />

treatment groups also received:<br />

Materials Group: The 66 teachers in the Materials group received the MTP language and literacy<br />

curriculum. The curriculum was provided via a “limited features” version of the MTP website.<br />

Web Group: The 89 teachers in the Web group received access to the full-featured MTP website, including<br />

the curricula described above, plus video demonstrations, sample lesson plans, and professional<br />

development activities, among other things.<br />

Consultancy Group: The 80 teachers in the Consultancy group received all of the above features and also<br />

collaborated with an MTP consultant to improve their teaching practice (via online chat and by using the<br />

MTP website to review their videos and respond to consultant-driven prompts)<br />

Multidimensional data are being collected on the effects of MTP, including child, teacher, and classroom quality<br />

outcomes.<br />

Child Outcomes<br />

In each of the two intervention years of our project, four children were randomly selected from each teacher’s<br />

classroom and assessed (pre and post) on language and literacy. (With attrition due in large part to families<br />

moving away from participating school districts, we are now following a total of 1,659 children.) Pre-K teachers<br />

and videotaped classroom observations are providing information on children’s social relationships and selfregulation,<br />

in addition to literacy and language development. All children are being followed into kindergarten<br />

and first grade with teacher questionnaires assessing language, literacy, and social relationships/regulation.<br />

We hypothesize an additive effect for the Web and Consultancy treatments, when combined with the MTP<br />

teaching materials, and anticipate that children will show progressively greater growth in child outcomes through<br />

first grade.<br />

Teacher & Classroom Quality Outcomes<br />

Observations of teaching practice and classroom quality were collected via videotape during the two intervention<br />

years and are now being collected in a third non-intervention follow-up year. Observations are coded using the<br />

Classroom Assessment Scoring System (CLASS), a coding system for the 14 different dimensions of teaching<br />

201


quality that are the focus of the MTP web-based resources. Studies involving over 2,000 pre-kindergarten and K-<br />

1 classrooms have shown that higher ratings on the dimensions assessed by the CLASS predict higher<br />

performance of children on standardized assessments of academic achievement and better social adjustment in<br />

the early grades of school (La Paro, Pianta, & Stuhlman, 2004). The relationship between the different aspects of<br />

this model is described in Figure 7.<br />

Figure 7. Linking Professional Development with teacher and child outcomes through standardized observations<br />

of classroom processes (Pianta, et al., 2004)<br />

The child and teacher outcomes from this project will be the subject of forthcoming papers. In addition, we are<br />

developing a methodology for evaluating the reflectiveness of teachers participating in the consultancy, and<br />

expect to examine the evolution of teacher reflections over time concurrently with teaching quality.<br />

Evaluation Outcomes<br />

It should be noted that even with the best of design and implementation methods (we provided a 1.5 day<br />

introductory teacher workshop, along with extensive on-line technical reference materials and FAQs), teachers<br />

still require substantial technical support, and since teachers were from many different districts with different<br />

computer skills, their problems tended to be diverse. Our approaches to technical support evolved over the first<br />

year of the project, as support shifted away from being provided by teaching consultants to the technical staff<br />

using a structured support system (see Kelly, Whitaker, Neesen, Kinzie, & Pianta, 2005, for an overview). This<br />

resulted in a greater amount of support provided to teachers in the same amount of time, high levels of teacher<br />

satisfaction, and no dropouts due to technical challenges.<br />

There are obvious advantages to our technology-assisted process for content development and content review.<br />

For instance, drawing upon video submitted from participating teachers and employing an on-line video<br />

production and quality review process has meant that we can produce between five and ten demonstration videos<br />

in as little as six hours, a significant advantage for MTP when we have hundreds of teaching activities we would<br />

like to demonstrate (see Neesen, et al., 2005, for more on this approach to video production).<br />

Our user-centered design process has yielded a website perceived to be easy to use and worthwhile, judging from<br />

the feedback we received from teachers in our first implementation year: Overall, 92.7% of the 110 MTP<br />

teachers who responded to our mid-year evaluation survey “agreed” or “strongly agreed” that the MTP website<br />

was easy to use; 88.2% “felt that the site added value to their professional practice, and 87.2% indicated that<br />

using the MTP website was worth the time spent on it. With data from the end-of-year survey of 203<br />

participating teachers, we conducted analyses by treatment group (Materials [n = 45], Web [n = 79], Consultancy<br />

[n = 79]). Teachers in the Web and Consultancy groups, who received higher “doses” of MTP, reported higher<br />

perceptions of value and usefulness, and felt more strongly that MTP was worthwhile, than did the Materials<br />

group teachers (Whitaker, Kinzie, Kraft-Sayre, Mashburn, & Pianta, 2005). Teachers receiving different<br />

amounts of support also evidenced differing levels of participation (Whitaker, et al., 2005).<br />

According to server log analysis of the on-line behavior of teachers in the first implementation year, teachers in<br />

the Materials group [n = 50], whose version of the website contains only the curricula, visited the MTP site an<br />

average of only 6.4 times during the school year, but they spent significantly more time during each visit<br />

(average length of 10.9 minutes each time), perhaps because these teachers needed to print out the MTP<br />

activities they wanted to use. These Materials group teachers also reported spending the least amount of time<br />

preparing and delivering MTP activities. Web-group teachers [n = 79] accessed the site an average of 10 times<br />

each, with an average length of 4.2 minutes per visit. Consultancy teachers [n = 84] accessed the site<br />

202


significantly more often than the other two groups, an average of 40.8 times each, and they spent longer than<br />

teachers in the Web group, with a mean visit of 7.62 minutes (Whitaker, et al., 2005).<br />

Overall, teachers have been enthusiastic in their support of the program, making comments such as:<br />

“I would say this is the best professional development opportunity that I have had in my 23 years of<br />

teaching.”<br />

“My children’s assessment scores have never been higher…this year I have seen scores in many cases<br />

triple!”<br />

When asked about their favorite aspect of MyTeachingPartner, many teachers mentioned MTP professional<br />

development features:<br />

“Ease of using the lesson plans on the Internet.”<br />

“My favorite part of MTP is the activities and suggestions given on the website.”<br />

“I like viewing the videos for presenting the books and using the MTP activities.”<br />

“Having a consultant made me more reflective and focused on what I need to do each day.”<br />

These results support the adage: if you build it, they will come: Teachers voluntarily spent additional time<br />

preparing to deliver a new curriculum and participating in the professional development that MTP provides, and<br />

that they found it of significant value. With MTP, we were able to employ effective and efficient development<br />

methods, in order to provide teaching materials and professional support that 235 teachers valued and spent time<br />

using. Our future research will show the degree to which these efforts influence the development and school<br />

readiness of children who are at-risk.<br />

Acknowledgements<br />

This research was supported by a grant from the National Institute of Child Health and Human Development<br />

(NICHD), National Institutes of Health (NIH). A special thanks to the hard-working and talented staff of My<br />

Teaching Partner and our participating Pre-Kindergarten teachers across the commonwealth of Virginia, who<br />

make a very important difference to at-risk children.<br />

References<br />

Blank, H, Schulman, K. & Ewen, D. (1999). Seeds of Success, State Prekindergarten Initiatives, 1998-1999.<br />

Washington, DC: Children’s Defense Fund.<br />

Bowman, B. T., Donovan, M. S. & Burns, M. S. (2001). Eager to learn. Washington, DC: National Academy<br />

Press.<br />

Bryant, D., Clifford, R., Pianta, R. C., Howes, C. & Burchinal, M. (2002) (in press). Characteristics of prekindergarten<br />

programs in six-states: Children, teachers, and programs. Applied Developmental Science.<br />

Clifford, D. & Maxwell, K. (2002). The need for highly qualified prekindergarten teachers. Paper presented at<br />

the Preparing Highly Qualified Prekindergarten Teachers Symposium. Retrieved 31 August <strong>2006</strong>, from<br />

http://www.fpg.unc.edu/~npc/pdfs/need.pdf<br />

Greenberg, M. T., Kusche, C. A., Cook, E. T. & Quamma, J. P. (1995). Promoting emotional competence in<br />

school-aged children: The effects of the PATHS curriculum. Development and Psychopathology, 7, 117-136.<br />

Justice, L, Pullen, P., Hall, S., & Pianta, R. (2003). Curry School Curriculum for Early Literacy and Oral<br />

Language Support. Unpublished. University of Virginia, Charlottesville.<br />

Justice, L. M. & Kaderavek, J. (2004). Embedded-explicit emergent literacy I: Background and description of<br />

approach. Language, Speech, and Hearing Services in Schools, 35, 201-211.<br />

Howes, C., Burchinal, M., Pianta, R., Bryant, D., Early, D., Clifford, R. & Barbarin, O. (in press). Ready to<br />

learn? Children's pre-academic achievement in pre-kindergarten programs. Developmental Psychology.<br />

203


Kelley, M. A., Whitaker, S. D., Neesen, K., Kinzie, M. B. & Pianta, R. C. (2005). Tech support requirements for<br />

large-scale technology innovation in teacher professional development. Paper presented at the annual meeting<br />

of E-Learn, Vancouver.<br />

La Paro, K., Pianta, R. & Stuhlman, M. (2004).Classroom Assessment Scoring System (CLASS): Findings from<br />

the Pre-K Year. Elementary School Journal, 104 (5), 409-426.<br />

Neesen, K., Kinzie, M. B., Whitaker, S. D., Funk, G. G., Hall, A. P. & Pianta, R. C. (2005). Educational video<br />

production to support teacher reflection and professional development: A high quality, cost effective<br />

collaboration between educators, researchers and educational technologists. Paper presented at the annual<br />

meeting of E-Learn, Vancouver.<br />

NICHD Early Child Care Research Network (ECCRN). (2002). Structure>Process>Outcome: direct and indirect<br />

effects of caregiving quality on young children’s development. Psychological Science, 13, 199-206.<br />

Pianta, R. & Hamre, B. (2001). Students, Teachers, and Relationship Support (STARS). Lutz, FL: Psychological<br />

Assessment Resources, Inc. Available at www.parinc.com<br />

Pianta, R. C. (1999). Enhancing relationships between children and teachers. Washington, DC: American<br />

Psychological Association.<br />

Pianta, R. C., Hall, A. P., Dudding, C., Whitaker, S., Kraft-Sayre, M. & Downer, J. (2004). MTP Procedural<br />

Manual for Teachers. Center for Advanced Study of Teaching and Learning, University of Virginia.<br />

Snow, C. E., Hamphill, L., & Barnes, W. S. (Eds.). (1991). Unfulfilled expectations: Home and school influences<br />

on literacy. Cambridge, MA: Harvard University Press.<br />

Vernon-Feagans, L. (1996). Children’s talk in communities and classrooms. Cambridge, MA: Blackwell.<br />

Whitaker, S. D., Kinzie, M. B., Kraft-Sayre, M. & Pianta, R. C. (2005). Use and Evaluation of Web-based<br />

Professional Development Services Across Level of Service and By Teacher/District Characteristics. Paper<br />

presented at the annual meeting of E-Learn, Vancouver.<br />

204


Hernández-Ramos, P. (<strong>2006</strong>). How Does Educational Technology Benefit Humanity? Five Years of Evidence. Educational<br />

Technology & Society, 9 (4), 205-214.<br />

How Does Educational Technology Benefit Humanity? Five Years of<br />

Evidence<br />

Pedro Hernández-Ramos<br />

Dept of Education & Center for Science, Tech, and Society, Santa Clara University, CA 95053 USA<br />

Tel: +1 408 554-4131<br />

Fax: +1 408 554-2392<br />

phernandezramos@scu.edu<br />

ABSTRACT<br />

This article presents a review of the 25 finalists (Laureates) in the Education category of the Technology<br />

Benefiting Humanity Awards, which started in 2001. Most of the applicants can be considered social<br />

entrepreneurs working to improve educational systems and the learning opportunities and experiences of<br />

their intended beneficiaries. While the benefits to humanity derived from educational technology are often<br />

taken for granted, the work of these 25 Laureates, working in countries around the world, provides an<br />

opportunity to critically examine how such benefits are actually obtained. Based on this analytical review,<br />

suggestions for further research and development in educational technology are presented.<br />

Keywords<br />

Social entrepreneurs, Educational technology, Development, Awards<br />

Introduction<br />

Two major reports published since 2000 highlight the importance of education for economic and social<br />

advancement in less-developed countries. The United Nations’ Millenium Development Goals (MDGs) set down<br />

on September 18, 2000 committed all 191 member states to meet eight major goals, with number two being:<br />

“Achieve universal primary education—ensure that all boys and girls complete a full course of primary<br />

schooling” (United Nations, 2005). Similarly, the World Education Forum that took place in Senegal also in<br />

2000 established six goals (Basic Education Coalition, 2004) that, in a way, add detail to the UN’s Millenium<br />

Development Goals regarding education:<br />

1. Expanding and improving comprehensive early childhood care and education, especially for the most<br />

vulnerable and disadvantaged children.<br />

2. Ensuring that by 2015 all children, particularly girls, children in difficult circumstances, and those belonging<br />

to ethnic minorities, have access to and complete free and compulsory primary education of good quality.<br />

3. Ensuring that the learning needs of all young people and adults are met through equitable access to<br />

appropriate learning and life skills programs.<br />

4. Achieving a 50% improvement in levels of adult literacy by 2015, especially for women, and equitable<br />

access to basic and continuing education for adults.<br />

5. Eliminating gender disparities in primary and secondary education by 2005, and achieving gender equality<br />

in education by 2015, with a focus on ensuring girls’ full and equal access to and achievement in basic<br />

education of good quality.<br />

6. Improving all aspects of the quality of education and ensuring excellence so that recognized and measurable<br />

learning outcomes are achieved by all students, especially in literacy, numeracy, and essential life skills.<br />

Given these goals, the question arises as to why should poor, less developed countries spend scarce resources<br />

providing technology (usually computers and Internet access) to only a few of their schools when most other<br />

schools lack even the basics of proper buildings, enough and properly trained teachers, books, and other learning<br />

materials? Also, the fact that technology-related projects tend to be quite expensive and usually financed by<br />

foreign loans that are open to graft and misuse (Easterly, 2001) gives some people reasons to advocate against<br />

such projects.<br />

In a developed world context like the U.S. Japan, Australia, or Europe, the reasons for introducing technology<br />

into education seem obvious if they are rooted in goals like the overall improvement of teaching and learning:<br />

introducing technology to schools helps the system to better prepare students for their future, when they’ll need<br />

so-called 21 st century skills (e.g., Fulton, 1997). In developing world contexts, however, where four fifths of the<br />

world population lives, the desired benefits of educational technology often cannot be focused on improving on<br />

existing practices as much as in creating practices where there may be none—or none worth holding on to. The<br />

motivation often comes from a belief in the potential benefits of technology to allow education to reach unserved<br />

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the<br />

copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies<br />

are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by<br />

others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior<br />

specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.<br />

205


or underserved populations. For example, the United Nations Development Program’s (2003) Human<br />

Development Report argued that “Technological innovations advance human development in two ways—by<br />

increasing productivity that raises household incomes (Goal 1 [of the MDGs]) and by providing solutions to<br />

problems of disease, transport, energy, water supply, sanitation and information and communications technology<br />

for education, all important for achieving Goals 2–7 [of the MDGs]” (p. 158).<br />

On the other hand, there is also a body of literature that questions the commitment to technology in the future of<br />

education (e.g., Cuban, 2001; Ferneding, 2003). Part of the challenge comes from understanding the profound<br />

structural changes that the integration of technology requires of education systems: financial, social, cultural, and<br />

of course pedagogical. The “failures” of past attempts to integrate technology into teachers’ daily practice (e.g.,<br />

Cuban, 1986) also serve as an argument used by critics to request or demand a change in directions away from<br />

technology investments (mainly computers and Internet access), and by extension, away from many of the<br />

pedagogical practices associated with the use of technology for teaching and learning. The point made for<br />

education specifically can be generalized for national economic and social development, as the Markle<br />

Foundation’s report, Global Digital Opportunities (Tipson & Frittelli, 2003), argued by stating that, “Simply<br />

increasing connectivity or distributing computers and software will not lead to development impacts unless the<br />

range of other factors—enterprise, applications, human capacity, policy—converge to make that possible”<br />

[emphasis in original] (p.8).<br />

The fact that technologies by themselves are incapable of achieving widespread and deep changes in education is<br />

the key argument offered by critics of educational technology. It happens to be, as well, a point easily<br />

acknowledged by even the most ardent proponents of the use of technology in education (e.g., Fisher, Dwyer, &<br />

Yocam, 1996; Osin, 1998). Tipson & Frittelli (2003) summarized it neatly by stating that<br />

ICTs [Information and Communication Technologies] clearly have major contributions to make<br />

in education, healthcare, gender equality, environmental sustainability and the other MDGs, but<br />

to be successful, that contribution should always be viewed as dependent on a range of other<br />

factors and subject to comparisons with other, less technology dependent approaches (p.13).<br />

The balance of this article will focus on the work of the first 25 Laureates (2001–2005) in the Education category<br />

of the Technology Benefiting Humanity Awards (http://www.techawards.org/), most of whom are operating in<br />

less developed countries. A major goal of this awards program is to recognize and motivate the work of<br />

innovators around the world who are working to benefit humanity (“social entrepreneurs”) through the<br />

development or application of technology. This Awards program has quickly established itself as a global<br />

positive force for change and innovation, attracting hundreds of applications. The five Education Laureates<br />

selected each year come from among dozens of applicants (over 100 each year so far). An international judging<br />

panel selects only one Education Laureate each year to receive a prize of US$50,000. A brief summary of their<br />

work is presented next, followed by analysis of the patterns and trends that can be deduced from the range of<br />

their activities. It will end with conclusions and recommendations for future actions and research.<br />

The Education Laureates<br />

Cardwell (1995) wrote that, “All that can be said is that it would be unwise to assume that all or even most<br />

innovators are motivated by purely mercenary hopes. Personal satisfaction and the hope of social distinction are,<br />

no doubt, factors in the lives of many technologists as well as in the lives of many ‘pure’ scientists” (p. 495).<br />

This definition of an innovator certainly applies to social entrepreneurs, which in turn have been defined as,<br />

“One who has created and leads an organization, whether for-profit or not, that is aimed at creating large scale,<br />

lasting, and systemic change through the introduction of new ideas, methodologies, and changes in attitude”<br />

(Kramer, 2005, p.6).<br />

The five Education Laureates in each of the first five years certainly would fit that description, since their work<br />

seems motivated more by the desire to improve the human condition than financial gain. Table 1 presents a<br />

summary of the problem and solution(s) offered by each Laureate. (See also Hernández-Ramos, Soukup, &<br />

McAnany, 2001; Soukup, 2002; Hernández-Ramos, 2003; Raphael, 2004; and Raphael, 2005.)<br />

Table 1. Summary of Problems Identified and Solutions Offered by Education Laureates<br />

Year Laureate Problem Solution(s)<br />

2001 Freeplay Foundation Lack of access to education, Windup radios and programming<br />

health information; cost of aimed at populations in critical<br />

206


atteries situations in Africa<br />

2001 Kids’ Space The Internet as an unorganized<br />

resource for children’s learning<br />

2001 Lewis Center for<br />

Educational Research<br />

activities<br />

How to make best use of a<br />

world-class telescope and<br />

scientific facility for teaching<br />

and learning in K-12 schools<br />

2001 Project Gutenberg How to disseminate thousands<br />

of “famous and important<br />

texts” in the public domain<br />

2001 Schools Online How to get schools in<br />

developing countries, to benefit<br />

from computers and the<br />

Internet.<br />

2002 Barefoot College How to offer educational<br />

options for poor people in<br />

remote parts of India<br />

2002 Global SchoolNet How to leverage computers and<br />

the Internet to expand the<br />

horizons of students<br />

2002 Center for Spoken<br />

Language Research<br />

everywhere in the world<br />

How to support learning for<br />

profoundly deaf children and<br />

others suffering from autism,<br />

dyslexia, and other disorders.<br />

2002 CAST How to best address the<br />

learning and reading needs of<br />

students with physical or<br />

cognitive disabilities.<br />

2002 Katha How to promote literacy,<br />

preserve culture, and encourage<br />

education among New Delhi’s<br />

poor.<br />

2003 Equal Access How to reach populations in<br />

remote places of developing<br />

countries (starting in Nepal) to<br />

provide information and<br />

education<br />

2003 Omar Dengo Foundation How to reform the education<br />

system while introducing<br />

educational technology<br />

2003 Computers for Youth How to motivate and engage<br />

youth and their families in lowincome<br />

middle schools<br />

2003 Brij Kothari-IIM How to help millions of people<br />

in India with no access to<br />

2003 Alexander McDonald,<br />

NOAA<br />

schools to learn to read<br />

How to convey complex timebased<br />

data on the human<br />

impact on the Earth, in<br />

meaningful ways, to diverse<br />

Moderated website with activities<br />

specifically designed for school-<br />

age children<br />

Allow use of the radio-telescope<br />

and provide teaching and learning<br />

resources for science education and<br />

research.<br />

Website has over 5,000 texts in<br />

various electronic formats and<br />

languages available for free<br />

download.<br />

Create teacher training resources<br />

and establish strategic partnerships<br />

for effective local implementation.<br />

Network of 20 campuses in 13<br />

states trains people to create and<br />

maintain technology infrastructures<br />

for community benefit.<br />

Create resources and projects that<br />

teachers and students can use in<br />

their classrooms and share around<br />

the world.<br />

Developed computer-based,<br />

personalized, learning tools for<br />

learners with speech learning<br />

difficulties resulting from deafness,<br />

autism spectrum disorders,<br />

dyslexia, and others.<br />

“Thinking Reader” technology<br />

supports a wide range of learning<br />

modalities and disabilities.<br />

“Computer clubhouse,”<br />

community access, teacher<br />

training, education, and other<br />

services for children and women<br />

living in slums.<br />

Use Digital Satellite Radio and<br />

partner with local organizations to<br />

create and disseminate relevant<br />

content for hard-to-reach<br />

communities.<br />

Re-think teaching and learning to<br />

better prepare students and support<br />

teachers in the systematic<br />

transformation of classroom and<br />

community practices.<br />

Provide a computer with Internet<br />

access for each student, and<br />

involve them in relevant activities<br />

for the school and the community.<br />

Add subtitles to popular television<br />

shows to build the literacy skills of<br />

non-readers<br />

Create the “Science on a Sphere”<br />

3-D projection system that makes<br />

it possible to visualize complex<br />

data sets on a spherical surface.<br />

207


2004 Baruch College Computer<br />

Center for Visually<br />

Impaired People and<br />

Touch Graphics<br />

audiences<br />

How to help visually impaired<br />

learners deal with visual<br />

information.<br />

2004 DonorsChoose Address the school funding<br />

crisis in the U.S.<br />

2004 I*EARN How to leverage the power of<br />

computers and<br />

telecommunications for<br />

improved education and<br />

2004 Andrew E. Lieberman,<br />

Asociación Ajb’atz’<br />

Enlace Quiché<br />

2004 National Society for<br />

Earthquake Technology<br />

2005 Gilberth Clark, Telescopes<br />

in Education Foundation<br />

learning.<br />

How to counteract the decline<br />

of indigenous cultures and<br />

languages in Guatemala<br />

How to increase the knowledge<br />

and earthquake awareness of<br />

builders in Nepal<br />

How to offer students<br />

opportunities to work with<br />

world-class telescopes from<br />

school<br />

2005 Design that Matters, Inc. Teaching adults how to read in<br />

areas with few books and no<br />

electricity<br />

2005 Fahamu–Networks for<br />

Social Justice<br />

How to train staff of human<br />

rights organizations working in<br />

Africa<br />

2005 In2Books How to encourage the<br />

development of literacy skills<br />

among low-income students in<br />

the U.S.<br />

2005 MIT Open CourseWare How to share educational<br />

resources in ways that protect<br />

intellectual property yet<br />

enhance the knowledge<br />

commons<br />

Develop the Talking Tactile Tablet<br />

that supports access to graphical<br />

and multimedia applications.<br />

Set up a website that matches<br />

teacher-identified needs with<br />

potential donors.<br />

Online forums, project-based<br />

learning, and other activities<br />

connecting teachers and students<br />

around the world.<br />

Develop a series of multimedia and<br />

web-based materials, and other<br />

community services, that preserve<br />

and enhance local indigenous<br />

cultures.<br />

Created the “Shake Table” to<br />

demonstrate the effects of<br />

earthquakes on buildings erected<br />

using inadequate vs. modern<br />

building techniques.<br />

Web-based access and control of<br />

sophisticated telescopes in the<br />

U.S., Australia, and Chile.<br />

The Kinkajou projector relies on<br />

solar batteries, high-efficiency<br />

LEDs, low-cost optics in a<br />

projector that shows print and<br />

images from microfilm.<br />

Created custom-developed courses<br />

delivered via workshops, CD-<br />

ROMs, and email to serve their<br />

audience.<br />

Literacy program gives 5 books to<br />

each participating child, and pairs<br />

each student with an adult “pen<br />

pal” via letters and email<br />

correspondence.<br />

Launched the MIT Open<br />

CourseWare initiative in 2002,<br />

where information and resources<br />

for the majority of MIT courses are<br />

available free of charge.<br />

As can be deduced from the list in Table 1, there have been no corporations, only one government organization<br />

(Alexander McDonald is affiliated with the U.S. National Oceanographic and Atmospheric Administration), and<br />

only three individuals (Kothari and McDonald in 2003, Clark in 2005) so far among the Laureates. While the list<br />

seems weighted toward organizations from the United States, many of them were being recognized for work<br />

done outside of this country or with a deliberately global scope.<br />

Whether or not the Laureates were aware of the United Nations’ MDGs and the six goals of the World Education<br />

Forum, their work contributes to the achievement of those goals in measurable ways. From 2001, for example,<br />

what FreePlay Foundation and Schools Online were doing addresses directly the need of providing access to<br />

information and education through technology (wind-up radios for FreePlay, computers with Internet access for<br />

Schools Online) for hard-to-reach populations.<br />

208


It is also difficult—and to some extent meaningless—to attempt to categorize the work of the Laureates into<br />

narrow niches, such as saying that solutions are hardware, software, or services since in most cases the work was<br />

recognized precisely because it is not just a technology but a systemic, socio-technical approach that<br />

encompasses other critical elements for success. A clear example of this is Costa Rica’s Omar Dengo Foundation<br />

(2003 Laureate), which not only provides hardware and software for schools throughout the country but is also<br />

in charge of teacher training, community outreach, and other essential services needed to make the introduction<br />

of technology a success (Ringstaff & Kelly, 2002). The pattern makes clear that one of the criteria the Tech<br />

Awards judges use is not just how impressive for its own sake the technology in use may be, but also the factors<br />

surrounding its implementation in the intended contexts for the target audiences. There is an appreciation for the<br />

value of designing systems around tools to accomplish complex objectives. Given the contexts in which many of<br />

the Laureates work, simple technologies with good systems around them may be better than the latest<br />

technologies but with less support. A very good example of this approach is Fahamu (2005 Laureate), which<br />

relies on CD-ROMs, email, and in-person workshops to build capacity for human rights organizations given that<br />

the Web is simply not as accessible or affordable as these older, technically simpler media.<br />

Among these first 25 Laureates there are three outstanding examples of hardware and software technologies<br />

developed to address the unique learning needs of people with disabilities, whether physical or cognitive: Center<br />

for Spoken Language Research (2002 Laureate), Center for Applied Special Technology (CAST, 2002<br />

Laureate), and Baruch College Computer Center for Visually Impaired People and Touch Graphics (2004<br />

Laureate). Development of the hardware and/or software was only the starting point in all three cases. For<br />

example, CAST created the Thinking Reader software as well as a broader framework, the Universal Design for<br />

Learning, to help teachers differentiate their lessons so that the needs of all learners—including those with<br />

disabilities in integrated classrooms—are successfully met. Thus, CAST provides a critical scaffold (the software<br />

to support learners with reading difficulties) as well as the pedagogical framework within which the scaffolding<br />

technology is best used.<br />

Students with disabilities raise significant challenges for educators interested in affording them the same quality<br />

of learning experiences as “normal” students. These solutions tend to be initially expensive, which limits their<br />

dissemination, although it is not hard to imagine how all learners could eventually benefit from these advances.<br />

The analogy to curb cuts in sidewalks, initially seen as a benefit for wheelchair-bound individuals, is also a help<br />

to mothers pushing baby strollers, bicyclists, older people, and so on. This is an exciting prospect for education<br />

around the world, given that if the pattern of rapidly decreasing costs of new technologies continues (as it is<br />

likely to), disabled learners in countries other than those originating the technology (in these three cases, the<br />

United States) are likely to eventually benefit. One strategy being pursued is to encourage Laureates to enter into<br />

licensing agreements that extend the reach beyond the original developers to others who can include the<br />

technology in their products.<br />

Looking beyond from developed world contexts has the benefit of allowing us to see how practitioners in other<br />

parts of the world define and then address common problems in education such as funding, training, access to<br />

resources, and serving a wide variety of learners including those with physical or other learning disabilities.<br />

Also, it allows us to consider both formal and informal education settings, since the educational infrastructure of<br />

the formal systems (including inadequate funding, poor buildings, insufficiently trained teachers, resource<br />

deficiencies, and so on) some times makes it more efficient to address the problem through alternative strategies.<br />

Discussion<br />

The 20 Laureates share some distinguishing characteristics even across their widely different approaches to the<br />

integration of technology in education and learning. Perhaps among the most salient is the recognition that<br />

educational technology is not simply computers and the Internet, as much of the discourse in developed countries<br />

seems to have focused, but should adopt a broader view of technological innovation. Such a view goes beyond<br />

cutting-edge hardware and software to include older technologies and, critically, a holistic approach to the<br />

implementation of complex systems that recognize the social dimensions of technology-based innovations. On<br />

the other hand, those who have been recognized for truly innovative technological solutions have clearly<br />

identified how their approach is better than any alternatives available to the populations they aim to serve (e.g.,<br />

disabled learners).<br />

From a learning theory perspective, it is striking to recognize how many of these projects are working—<br />

explicitly or implicitly—from a constructivist perspective (e.g., Fosnot, 2005; Jonassen, Peck, & Wilson, 1999),<br />

to afford learners meaningful experiences connected to or in the “real world” and giving them access to the<br />

209


professional-level tools and resources of practicing scientists, researchers, and so on. Perhaps the best examples<br />

among these 25 Laureates are the Lewis Center for Educational Research (2001 Laureate) and Gilbert Clark of<br />

the Telescopes in Education Foundation (2005 Laureate). Each of these efforts allows teachers and their students<br />

access to professional-level tools and resources (a radio-telescope through the Lewis Center, and several remotecontrolled<br />

telescopes through Telescopes in Education). It’s easy to imagine the excitement felt by those students<br />

when given the chance to work in such a facilities, doing work as real scientists would. At another level, the<br />

work by Nepal’s National Society for Earthquake Technology (2004 Laureate) exemplifies the belief that people<br />

may learn best when they get to do and see and touch rather than just listen. Their Shake Table provides a<br />

powerful and direct way to instruct builders on the consequences of inadequate building techniques and<br />

materials, and thanks to the ability to simply and cheaply simulate how different structures fare in earthquakes of<br />

different magnitudes, turn what may seem as abstract concepts into memorable lessons.<br />

Considering first the approach to technology as meaning more than computers and the Internet, a closer look at<br />

the work of the Freeplay Foundation, the Barefoot College, Equal Access, and Brij Kothari’s Same Language<br />

Subtitling is rewarding. In each case, the technology used is not a breakthrough technology except in the context<br />

where it was being introduced, as in the regions of Africa where the Freeplay Foundation’s windup radios<br />

addressed a major problem of lack of access to information because the cost of batteries made operating a<br />

transistor radio too expensive. The Barefoot College in India broadens the definition of technology to include<br />

both hardware (solar panels for electricity generation) and software (work processes that empower local<br />

communities through harnessing of natural resources). Equal Access, on the other hand, relies on state-of-the-art<br />

digital satellite radio broadcast and receiving technology (including a custom-designed portable receiver<br />

intended for personal or small group use) to reach populations that cannot have access to education and<br />

information through any other media. And Brij Kothari’s elegantly simple idea to leverage subtitling of televised<br />

images (mainly the very popular music variety shows broadcast in Indian television) as a way to expose illiterate<br />

populations to the song lyrics—that is, the written languages they have had little or no exposure to in their<br />

limited schooling experience—thus supporting beginning literacy.<br />

Each of these four examples relied on technologies invented by someone else, but the Laureates made creative<br />

use of them to address specific needs they identified. Cardwell’s (1995, p.493) valuable distinction between<br />

machines (temporary, replaceable by better machines) and structures (designed to be permanent) is applicable<br />

here, for even the Laureates designing mainly hardware clearly understand the need to have appropriate<br />

structures around them that enhance the likelihood of successful, long-term implementation. The ripple effects of<br />

their work on the communities they serve are broad and deep, in some cases leading to significant changes in<br />

living standards and quality of life.<br />

Cases like the Costa Rican national educational technology implementation, headed by the Omar Dengo<br />

Foundation (2003 Laureate), highlight the importance of vision and leadership in change processes of this<br />

magnitude. “National leaders must advance a vision of change that can steer a political course that somehow<br />

balances rapid development with social cohesion and cultural integrity” (Tipson & Frittelli, 2003, p. 10), a<br />

challenge that can be easily biased in the direction of political self-interest and preservation of special interests.<br />

Costa Rica is, along with Chile (Hepp, Hinostroza, Laval, & Rehbein, 2004), one of the few success stories in<br />

Latin America where changes of this magnitude to national education systems have been accomplished,<br />

surviving political changes that destroy similar efforts in other countries. The process is not over, even after<br />

more than 15 years of sustained work in Costa Rica, a fact that serves to reminds us that change—and the<br />

benefits derived from it— can be painfully slow even under the best of circumstances.<br />

It’s also worth noting that most of the Laureates are working toward improvement of conditions of people at the<br />

bottom of the pyramid (Prahalad & Hammond, 2002). The systemic efforts to employ technology to address the<br />

needs of populations in dire need is evident in the work of organizations such as the Barefoot College (2002<br />

Laureate), Freeplay Foundation (2001 Laureate), Katha (2002 Laureate), Schools Online (2001 Laureate), Equal<br />

Access (2003 Laureate), Computers for Youth (2003 Laureate), Andrew Lieberman of the Asociación Ajb’atz’<br />

Enlace Quiché (2004 Laureate), and Design that Matters (2005 Laureate). In all these cases, the goal is both to<br />

focus on a specific population and then try to reach as many of them as possible. The technologies employed by<br />

these Laureates have been chosen wisely, to address the needs of the people served and enhance the<br />

multiplicative effect of the introduction of technology in their environment. In all these cases, technology is<br />

playing multiple roles, increasing individual opportunity and contributing to social cohesion and development.<br />

An underlying and often unstated assumption about ICTs is that “when designed and phased in with attention to<br />

each aspect of the strategic framework, ICTs should be able to trigger a form of development dynamic—a<br />

multiplier or “network effect” that generates an overall impact greater than the sum of the separate inputs”<br />

(Tipson & Frittelli, 2003, p.11).<br />

210


The experiences of these Laureates give credence to this assumption. Barefoot College, started in 1972, now has<br />

20 campuses in 13 states in India and is about to start training people from five other countries. Freeplay<br />

Foundation has distributed over 50,000 radios reaching an estimated one million people every day. Schools<br />

Online has installed computers with Internet access in over 400 schools in 35 countries, and trained hundreds of<br />

teachers in those schools to make the most of their new technology resources. The radio programs developed in<br />

cooperation with local partners by Equal Access, delivered through digital satellite radio, reach an estimated<br />

audience of 9 million people in Nepal, while in Afghanistan a broadcast aimed at teachers reaches an estimated<br />

3,500 teachers, benefiting the education of about 150,000 students. Katha in India and Asociación Ajb’atz<br />

Enlace Quiché in Guatemala are both agents of change and of cultural survival for the communities they serve,<br />

creating spaces and opportunities for people who would otherwise not know how to leverage the potential<br />

benefits of ICTs to learn better, express their creativity, and preserve their cultural heritage.<br />

Such “network effects” are even more in evidence when the technology in use is the Internet. Although based in<br />

the United States, what programs like Kids’ Space (2001 Laureate), Global SchoolNet (2002 Laureate), Project<br />

Gutenberg (2001 Laureate), and to some extent I*EARN (2004 Laureate) and MIT Open CourseWare (2005<br />

Laureate) are doing was simply not possible in a pre-Internet era. The ideals sustaining their work argue that<br />

ICTs are not meant to be merely “enablers” of outmoded but easily reproduced teaching and learning practices,<br />

but rather a force or catalyst for structural changes in education. By enabling access to rich resources and making<br />

possible communication and collaboration across time and space in well-structured virtual spaces, these<br />

Laureates have created wonderful opportunities for students and teachers who no longer have to be limited to the<br />

resources in their classroom and schools. Global SchoolNet and I*EARN, for example, each have thousands of<br />

schools and teachers participating in collaborative projects from over 100 countries around the world, involving<br />

more than one million students in learning experiences made all the more memorable because of the<br />

international connections. The MIT Open CourseWare initiative has already spawned similar projects at several<br />

other institutions in the U.S., Japan, and Spain, and also sparked a vital debate about the role of intellectual<br />

property in the future of education. At a time when intellectual property regimes seem to be becoming more<br />

restrictive, MIT’s attitude of generous information sharing has re-opened conversations about the academic<br />

values of free knowledge dissemination for the benefit of humanity at large.<br />

Concerns over the replicability, scalability, and sustainability of all these social benefit programs has led to the<br />

creation of the Global Social Benefit Incubator (GSBI) program (Koch, Coppock, Guerra, & Bruno, 2004;<br />

Mangan, 2004), which in addition to an intensive, two-week, on-campus workshop for selected attendees is<br />

actively investigating the use of online environments to support ongoing access to education, information,<br />

communication and other resources (Hernández-Ramos, Koch, Dommel, Guerra, & Bowker, 2005). The goal is<br />

to leverage online technologies to sustain and advance the work of social entrepreneurs around the world. The<br />

online environment would complement and extend the work that takes place during two-week summer<br />

workshops at a university campus, and will allow access to individuals and organizations that cannot participate<br />

in person.<br />

Conclusions<br />

Looking at the work of these first 25 Laureates of the Technology Benefiting Humanity Awards, it would be fair<br />

to conclude that “educational technology” can, indeed, benefit humanity. However, this general answer requires<br />

a deeper look at several common issues raised by this analysis and other experiences.<br />

First among these is the fact that it is unproductive to speak of educational technology as a single entity, and in<br />

particular to limit its meaning to computers and the Internet. Almost all of these Laureates used computer and<br />

Internet technologies in their work, but they clearly went beyond providing access to computers and the Internet<br />

to the recognition that the process of change in educational practices and systems requires systemic, long-term<br />

approaches.<br />

A second question addresses the invisible demands of technology—the fact that extensive infrastructures may be<br />

needed to allow computers and the Internet to function properly (or at all) in education settings around the world.<br />

The world may be steadily moving toward a knowledge-based economy where networked computers and other<br />

devices are critical tools, but increasingly this also means that electrical, telecommunications, educational, and<br />

other infrastructures need to be in place before countries can hope to effectively participate in such an economy.<br />

Projects like Schools Online, Barefoot College, and Computers for Youth highlight the need for—and the<br />

benefits of—a systemic approach that aims to have all the pieces under their control in place, and work with<br />

partners to address all other components.<br />

211


Third, with the exception of the Omar Dengo Foundation in Costa Rica, these Laureates are working on the<br />

periphery of education systems rather than at the core, and thus in some cases it is difficult to imagine them<br />

being able to scale up to a point where their impacts and influence could be systemic and go beyond a fairly<br />

narrow audience (Joyce, 2005). The positive impact of their mere presence should not be underestimated, but at<br />

the same time, as organizations they face the typical challenges of growth and sustainability if they are to survive<br />

and thrive for the long term. As mentioned above, the GSBI program intends to address this issue by developing<br />

the skills and knowledge bases of key members of the Laureates’ organizations, initially through on-campus<br />

training and later through an online environment.<br />

While not a representative sample of the universe of technological innovation in education around the world, this<br />

review does provide valuable information about possible directions for future research and development. These<br />

include:<br />

Electrical power<br />

The Freeplay Foundation’s development of the windup radio was a creative solution to vexing problems<br />

surrounding access to information through electronic media, which require electrical power to function. When a<br />

country’s electricity grid does not reach every community and household, and when portable power (batteries,<br />

fuel-powered plants) is too expensive, alternative sources of power (mechanical, wind, solar) that can keep<br />

equipment running for significant amounts of time, with little or no effort and at minimal cost, are a critical<br />

strategy to pursue. Literacy programs are among the first to benefit when future readers have easier access to<br />

electric lighting.<br />

Hardware<br />

Despite the relentless drops in price for computer and networking equipment, computers and peripherals<br />

designed to operate in industrialized office or home environments have a much shorter productive life when used<br />

in less-than-ideal circumstances, and are still too expensive for the vast majority of people around the world.<br />

Dust, heat, humidity, and unstable power sources are just some of the problems that may cause equipment to fail.<br />

There is ample room for creativity and innovation in this area, particularly if design efforts involve potential<br />

users from less-developed countries from the start as full-fledged co-designers, rather than merely as<br />

consumers—and even that, as an afterthought.<br />

Training<br />

Projects and programs from organizations like Schools Online, Global SchoolNet, I*EARN, the Lewis Center<br />

for Educational Research, Katha, the Omar Dengo Foundation, Computers for Youth, Enlace Quiché, and<br />

Fahamu spend considerable effort in preparing the intended beneficiaries to be successful over the long term.<br />

Learning and instructional technologies are rarely self-evident in their use, even at the operational level, and<br />

much less so when it comes to their integration into daily classroom (learning spaces) practice, highlighting the<br />

need for thoughtful training and ongoing professional development processes.<br />

Content<br />

While the Internet is, indeed, a wonderful resource for information and resources, much of what is available is in<br />

English and was created with an industrialized world audience in mind. Project Gutenberg, Equal Access, Katha,<br />

Brij Kothari’s subtitling technology, and Design that Matters’ Kinkajou project are examples of projects that<br />

allow free access to valuable content, much of it developed in close cooperation with representatives from the<br />

intended audiences. Affordable software tools that allow users with limited technical skills to create quality<br />

content in a variety of media (text, audio, graphic/photo, video, and multimedia), in their own and multiple<br />

languages, are needed to encourage and promote creativity as well as knowledge creation and dissemination at<br />

all levels.<br />

Structures and systems<br />

Most of the projects highlighted here, but particularly those based in or focusing their work in less-developed<br />

countries, would benefit from having ready access to communities of practice with whom to share information<br />

212


and resources that would enhance their work. In contexts where the lack of infrastructure means that getting<br />

anything done requires much greater effort, the opportunity to learn from the experiences of others to avoid<br />

making the same mistakes, and to do things better with the benefit of another’s experiences, would be a great<br />

benefit for all social entrepreneurs.<br />

Software<br />

In addition to the content-creation tools mentioned above, new software development tools that allow users and<br />

designers to easily create innovative simulations and other types of rich learning materials are needed. Just as<br />

CAST, the Center for Spoken Language Research, and the partnership between Baruch College’s Computer<br />

Center for Visually Impaired People and Touch Graphics have created wonderful software for learners with<br />

special needs, similar efforts addressing the needs of all other types of learners—and again, in a wide variety of<br />

learning contexts, in languages other than English only—are needed.<br />

Innovative hardware, software, systems, and organizational structures have the potential to bring the benefits of<br />

educational technology to the large majority of people around the world who don’t have access to even simple<br />

telecommunications (over half of the world’s population still has not made a telephone call; UNDP, 2003), and<br />

who live in countries where education and learning are mired in antiquated modes and structures that don’t serve<br />

their needs adequately or at all.<br />

As Tipson & Frittelli (2003) remarked, “Yet, enormous opportunities are being lost in the developing world from<br />

failure to capitalize on what is available in anticipation of what is (realistically) not” (p.8). One can easily<br />

imagine a utopian future where all social entrepreneurs are “out of business” because the world’s problems have<br />

been solved. In the meantime, the work of the 25 Education Laureates of the Technology Benefiting Humanity<br />

awards program must be recognized and encouraged.<br />

Epilogue<br />

The following Laureates were honored with the $50,000 prize: 2001—Freeplay Foundation; 2002—Barefoot<br />

College; 2003—Equal Access; 2004—Andrew Liberman of the Asociación Ajb’ atz’ Enlace Quiche; 2005—<br />

MIT Open CourseWare.<br />

References<br />

Basic Education Coalition. (2004). Teach a child. Transform a nation. Washington, DC: Author.<br />

Cardwell, D. S. L. (1995). The Norton history of technology. New York: Norton.<br />

Cuban, L. (1986). Teachers and machines. The classroom use of technology since 1920. New York: Teachers<br />

College Press.<br />

Cuban, L. (2001). Oversold and underused: Computers in the classroom. New York: Teachers College Press.<br />

Easterly, W. (2001). The elusive quest for growth. Economists’ adventures and misadventures in the tropics.<br />

Cambridge, MA and London: MIT Press.<br />

Ferneding, K. A. (2003). Questioning technology: Electronic technologies and educational reform. New York:<br />

Peter Lang Publishing.<br />

Fisher, C., Dwyer, D. & Yocam, K. (Eds.). (1996). Education & technology. San Francisco: Jossey-Bass.<br />

Fosnot, C. T. (Ed.) (2005). Constructivism: Theory, perspectives and practice. (2 nd . Ed.) New York: Teachers<br />

College Press.<br />

Fulton, K. (1997). Learning in a digital age: Insights into the issues. The skills students need for technological<br />

fluency. Los Angeles: Milken Exchange on Educational Technology.<br />

213


Hepp, P., Hinostroza, E., Laval, E. & Rehbein, L. (2004). Technology in schools: Education, ICT, and the<br />

knowledge society. Temuco, Chile: Instituto de Informática Educativa.<br />

Hernández-Ramos, P. (2003). The NASDAQ Stock Market Education Award. STS Nexus 4 (1), 16-20.<br />

Hernández-Ramos, P., Koch, J. L., Dommel, H. P., Guerra, P. & Bowker, G. (2005). Designing an online<br />

collaboratory for the Global Social Benefit Incubator. Paper submitted to the 2005 E-Learn Conference,<br />

Vancouver, BC, Canada.<br />

Hernández-Ramos, P., Soukup, P. & McAnany, E. (2001). Educational technology in the service of global<br />

leaders. NASDAQ Stock Market Education Award Finalists. STS Nexus (2) 1. Available online at:<br />

http://www.scu.edu/sts/nexus/fall2001/HernandezMcAnanySoukup.cfm.<br />

Jonassen, D., Peck, K. & Wilson, B. (1999). Learning with technology: A constructivist perspective. Upper<br />

Saddle River, NJ: Prentice Hall.<br />

Joyce, M. V. (2005). Scaling up success: Technology-based educational innovations— An HGSE news<br />

interview with Chris Dede and James Honan. HGSE News, May 1. Available at:<br />

http://gseweb.harvard.edu/news/features/scaling05012005.html.<br />

Koch, J. L., Coppock, K., Guerra, P. & Bruno, A. V. (2004). The global social benefit incubator: Toward a new<br />

pedagogy for “scaling” in social benefit entrepreneurship. STS Nexus 5 (1). Available online at:<br />

http://www.scu.edu/sts/nexus/fall2004/Koch-Coppock-Guerra-and-Bruno-Article.cfm.<br />

Kramer, M. R. (2005). Measuring innovation: Evaluation in the field of social entrepreneurship. Palo Alto, CA:<br />

Skoll Foundation/Foundation Strategy Group.<br />

Mangan. K. S. (2004). Teaching business skills to people with a social mission. Chronicle of Higher Education<br />

51 (10), 40a. Available online at: http://chronicle.com/weekly/v51/i10/10a01001.htm<br />

Osin, L. (1998). Computers in education in developing countries: Why and how. Education and Technology<br />

Series, Vol. 1 (3). Washington, DC: The World Bank Human Development Network-Education Group-<br />

Education and Technology Team.<br />

Prahalad, C. K. & Hammond, A. (2002). What works: Serving the poor, profitably. New York: Markle<br />

Foundation. Available online at: http://www.markle.org/news/What_works_Servingthepoorprofitably.pdf.<br />

Raphael, C. (2004). The Microsoft Education Award. STS Nexus 5 (1). Available online at:<br />

http://www.scu.edu/sts/nexus/fall2004/Raphael-Article.cfm.<br />

Raphael, C. (2005). The Microsoft Education Award. STS Nexus 6 (1), 28-34.<br />

Ringstaff, C. & Kelly, L. (2002). The learning return on our educational technology investment: A review of<br />

findings from research. San Francisco: WestEd.<br />

Soukup, P. (2002). The NASDAQ Stock Market Education Award. STS Nexus 3 (1). Available online at:<br />

http://www.scu.edu/sts/nexus/3.1fall2002/SoukupSJArticle.cfm<br />

Tipson, X. & Frittelli, C. (2003). Global Digital Opportunities. National Strategies for “ICT for Development.”<br />

New York: Markle Foundation.<br />

United Nations. (2005). UN Millenium Development Goals. Retrieved 24 August <strong>2006</strong> from<br />

http://www.un.org/millenniumgoals/index.html.<br />

United Nations Development Programme (UNDP). (2003). Human development report 2003. Millenium<br />

development goals: A compact among nations to end human poverty. New York and Oxford: Oxford University<br />

Press.<br />

214


Nevile, L., & Treviranus, J. (<strong>2006</strong>). Interoperability for Individual Learner Centred Accessibility for Web-based Educational<br />

Systems. Educational Technology & Society, 9 (4), 215-227.<br />

Interoperability for Individual Learner Centred Accessibility for Web-based<br />

Educational Systems<br />

Liddy Nevile<br />

La Trobe University, Bundoora, Victoria, Australia<br />

Tel: +61 4 1931 2902<br />

liddy@sunriseresearch.org<br />

Jutta Treviranus<br />

University of Toronto, Toronto, Ontario, Canada<br />

Tel: +1 416 978 5240<br />

jutta.treviranus@utoronto.ca<br />

ABSTRACT<br />

This paper describes the interoperability underpinning a new strategy for delivering accessible computerbased<br />

resources to individual learners based on their specified needs and preferences in the circumstances in<br />

which they are operating. The new accessibility strategy, known as AccessForAll, augments the model of<br />

universal accessibility of resources by engaging automated systems and builds upon the previous<br />

development of libraries of suitable resources and components. It focuses on individual learners and their<br />

particular accessibility needs and preferences. It fits within an inclusive framework for educational<br />

accommodation that supports accessibility, mobility, cultural, language and location appropriateness and<br />

increases educational flexibility. Its effectiveness will depend upon widespread use that will exploit the<br />

‘network effect’ to increase the content available for accessibility and distribute the responsibility for the<br />

availability of accessible resources across the globe. Widespread use will depend upon the interoperability<br />

of AccessForAll implementations that, in turn, will depend on the success of the four major aspects of their<br />

interoperability: structure, syntax, semantics and systemic adoption.<br />

Keywords<br />

E-learning systems, Interoperability, Accessibility, AccessForAll, Learner profiles, Resource descriptions<br />

Introduction<br />

This paper describes the effort to give effect to the interoperability goals of a new strategy for delivering<br />

accessible computer-based resources to learners based on their immediate specific needs and preferences. There<br />

are many reasons why learners have different needs and preferences with respect to their use of digital resources,<br />

including because they have disabilities. Instead of classifying people by their disabilities, the ‘AccessForAll’<br />

approach emphasizes the resulting needs in an information model for formal structured descriptions of those<br />

needs and the resources available. It provides a common language for describing the needs and preferences of all<br />

people, for whatever reason their accessibility is hampered, when they have special requirements. It may be<br />

useful to those who are simply trying to gain access to information in a language other than one they<br />

comprehend. It will be essential for those with very restricted access capabilities.<br />

AccessForAll uses the same formal, structured information model for describing the needs and preferences of<br />

users and the accessibility characteristics of resources. Both sets of descriptions are required so resources can be<br />

matched to the needs and preferences of the learner. The goal is to enable systems to share the benefits and<br />

burdens of making resources accessible. It must be easy to record the necessary information and it must be in a<br />

form that will make it most useful and interoperable.<br />

There is no doubt that an important aspect of achieving interoperability is the widespread adoption of common<br />

solutions to problems. The new framework aims, where possible, to inherit this from extensively used<br />

specifications and standards. In addition, it will depend on the widespread availability of suitable resources,<br />

appropriately described. In the case of educational resources and services, there are many major communities<br />

concerned with relevant aspects of descriptive standards and of those, a number have been engaged in the<br />

development of the AccessForAll model. Cross-domain metadata also has well-established standards that have<br />

been considered. The model is based on a set of principles that, when implemented in a variety of standard<br />

languages or systems, should maintain their interoperability at structural, syntactic and semantic levels. It also<br />

depends upon widespread systemic adoption to generate the required volume of information about available<br />

accessible components.<br />

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the<br />

copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies<br />

are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by<br />

others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior<br />

specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.<br />

215


The AccessForAll strategy complements work about how to make resources as accessible as possible<br />

(universally accessible) done primarily by the World Wide Web Consortium Web Accessibility Initiative<br />

(W3C/WAI). The focus of that work is technical specifications for the representation and encoding of content<br />

and services, to ensure that they are simultaneously accessible to as many people as possible. W3C also develops<br />

protocols and languages that become industry standards to promote interoperability for the creation, publication,<br />

acquisition and rendering of resources.<br />

Similarly, the strategy complements the work of organizations such as the Royal National Institute for the Blind<br />

(RNIB), The Library of Congress National Library Service for the Blind and Physically Handicapped (NLS), the<br />

Speech-to-Text Services Network and similar institutions in other parts of the world.<br />

In the United Kingdom, the RNIB developed and maintains the National Union Catalogue of Alternative<br />

Formats (NUCAF) for people with vision disabilities (Chapman, 2002). The American NLS maintains a Union<br />

Catalogue (BPHP) and a file of In-Process Publications (BPHI) that can both be searched via the NLS website.<br />

The Speech-to-Text Services Network (STSN) makes accessible content alternatives for content that cannot be<br />

used by people with hearing disabilities. They offer three types of services based on the technology used: Steno<br />

machine-based systems, commonly called CART (Communication Access Realtime Translation), Laptop-based<br />

speed typing software systems (C-Print and TypeWell) and Laptop-based Automatic Speech Recognition<br />

software systems (e.g., CaptionMic, iCommunicator).<br />

Such organizations provide access to resources through catalogues but as most of them pre-dated the Web, and<br />

certainly the widespread use of metadata as it is used today, their records are not in a common form and they do<br />

not all contain the same information in the same structure or syntax. Their catalogues provide search facilities of<br />

the type upon which the new AccessForAll strategy needs to be built.<br />

The focus of AccessForAll is ensuring that the composition of resources, when delivered, is accessible from the<br />

particular learner’s immediate perspective. It complements the W3C work by enabling a situation where a<br />

particular suitable resource is discoverable and accessible to an individual learner even when it may not be<br />

accessible to all learners. In some cases, this may mean discovery and provision of alternative, supplementary or<br />

additional resource components to increase the accessibility of an original resource. In such a case, those<br />

institutions that already have suitable resources may be able to lead to their discovery. The distinguishing feature<br />

of AccessForAll is that it depends upon software to manage the process of determining if the resources are<br />

suitable and, if not, replacing or augmenting them with the necessary resources. It assembles distributed,<br />

sometimes cumulatively-created, content into accessible resources and so is not wholly dependent upon the<br />

universal accessibility of the original resource and goes farther than mere discovery of a suitable resource.<br />

The AccessForAll specifications, while initiated in the educational community, are suitable for any user in any<br />

computer-mediated context. These contexts may include e-government, e-commerce, e-health and more. Their<br />

use in education will be enhanced if they are adopted across a broad range of domains and used to describe the<br />

accessibility of resources available to be used in education even if that was not their initial purpose. The<br />

AccessForAll specifications can be used in a number of ways, including: to provide information about how to<br />

configure workstations or software applications; to configure the display and control of on-line resources; to<br />

search for and retrieve appropriate resources; to help evaluate the suitability of resources for a learner, and in the<br />

sharing and aggregation of resources.<br />

The AccessForAll specifications are designed to gain extra value from what is known as the ‘network effect’: the<br />

more accessible content becomes available, the more likely accessible components will be available for<br />

individual learners; the more people use the specifications, the more there will be opportunities for interchange<br />

of resources or resource components, and the more opportunities there are, the more accessibility there will be<br />

for learners.<br />

Early versions of the specifications were developed by the Adaptive Technology Resource Centre (ATRC), at<br />

the University of Toronto. They were developed further by collaboration between IMS Global Learning<br />

Consortium; the Dublin Core Metadata Initiative Accessibility Working Group, and others. They have been<br />

considered formally by CEN ISSS MMI-DC, CEN ISSS (WS-LT) and ISO JTC1 SC36 as well as at the national<br />

level by government agencies in Canada, the UK, and Australia.<br />

Provision of Networked Accessible Resources<br />

There are a number of approaches to making networked resources accessible, whether on the Internet or on an<br />

Intranet.<br />

216


The first and most common approach is to aim to create a single resource (Web site, Web application) that meets<br />

all the accessibility requirements. Such a resource is known as a universally accessible resource. While this<br />

approach would work well in many situations, it is not often that authors succeed in making their resources<br />

‘universally accessible’, especially when they contain interactive components. As well, it is possible to make a<br />

resource that satisfies the guidelines but is, in fact, not accessible to everyone. There are also potential problems<br />

due to lack of attention to usability principles that may account for lack of satisfactory access (Disability Rights<br />

Commission (UK), 2004). The DRC rigorously tested sites that had passed the W3C accessibility tests by asking<br />

people with disabilities to use them and, in this process, followed standard usability testing techniques.<br />

In their report, the DRC said that:<br />

2.2 Compliance with the Guidelines published by the Web Accessibility Initiative is a necessary<br />

but not sufficient condition for ensuring that sites are practically accessible and usable by<br />

disabled people. As many as 45% of the problems experienced by the user group were not a<br />

violation of any Checkpoint, and would not have been detected without user testing.<br />

and<br />

FINDING 5: Nearly half (45%) of the problems encountered by disabled users when attempting<br />

to navigate websites cannot be attributed to explicit violations of the Web Accessibility Initiative<br />

Checkpoints. Although some of these arise from shortcomings in the assistive technology used,<br />

most reflect the limitations of the Checkpoints themselves as a comprehensive interpretation of<br />

the intent of the Guidelines.<br />

(The authors of the guidelines, W3C, have recognized these short-comings and are in the process of producing a<br />

revised set, Web Content Accessibility Guidelines Version 2 (W3C/WAI WCAG 2.0), that should better match<br />

their intentions.)<br />

Indeed, a resource may be accessible to everyone, but optimal for no one. Unfortunately, authors concerned<br />

about accessibility often avoid resource components that may be very effective, entertaining or efficient for some<br />

learners, for fear they will not be accessible to all learners (Macromedia’s Flash application is a typical<br />

example).<br />

The second approach used by a number of educational content providers is to create two versions of a resource: a<br />

media rich version and an “accessible version,” the latter being stripped of all media that may cause accessibility<br />

problems. Typically these alternative versions are available as ‘text only’ versions for selection by a user.<br />

UsableNet have developed a Lift Text Transcoder (UsableNet) that dynamically produces text versions of pages.<br />

This was done to avoid the mismatch of maintenance of pages where the pages were not automatically generated<br />

and content providers would up-date their media-rich pages but forget to do the same for their text-only versions.<br />

In many cases, media-rich pages are made available as text-only pages by conversion of the text and thus a lot of<br />

valuable content is simply not available to those who cannot access the media objects.<br />

While the provision of text-only pages solves some of the problems with the first approach, it can also cause<br />

other problems. In some cases, learners with disabilities only get an out-of-date view of the information but more<br />

often, students who perhaps need more assistance get less because they are using an impoverished version of the<br />

resource. The idea that learners with disabilities are a homogenous group that is well served by a single bland<br />

version of a resource is flawed.<br />

The third approach differs from the first two in a number of ways. Accessibility requirements are met not by a<br />

single resource but by a resource system. Rather than a single resource or a choice between two resource<br />

configurations, there can be as many configurations as there are learners. The ability of the computer mediated<br />

environment to transform the presentation, change the method of control, to disaggregate and re-aggregate<br />

resources and to supplement resources is capitalized upon to match resource presentation, organization, control<br />

and content to the needs of each individual learner. The availability of automated services as described makes it<br />

feasible for all learners to consider their needs and preferences and to record them for use by such a system. An<br />

approach that is inclusive in this way significantly increases the multiplicity of presentations available from a<br />

given source and thus shifts the responsibility for personalisation of presentations from the content provider, as<br />

designer of presentations, to the user. It also decreases the effort required on the part of individual content<br />

creators to anticipate the necessary combinations. It is an approach that lets the user select their desired content<br />

rather than one that depends on the website designers’ best guesses at the options that may be required.<br />

Extending the range of presentations of content as proposed by the third approach, the AccessForAll approach<br />

literally means moving towards mass customization of content provision. It is not merely relevant in the context<br />

217


of people with special needs, or disabilities, but for everyone. It is applicable in a range of contexts as it will lead<br />

to highly personalized learning experiences, on demand learning and content assembly and publishing,<br />

automated generation of content, and more. The availability of the information about the needs and preferences<br />

of the user is the key feature here. Many existing systems provide descriptions of content, and many provide<br />

customized assemblies, but the new approach provides for these to be user driven.<br />

The notion of universality of personalization, of inclusiveness as a characteristic of learning resources for<br />

everyone, differentiates the AccessForAll approach from the more traditional universal accessibility of resources<br />

approach, described above. The goal is not new, and has been considered many times, but it is contended that the<br />

new approach to user needs and preferences management actually makes it possible and, perhaps then, a<br />

necessary response to the needs of learners if all learners are to be catered for equally.<br />

The possibility is not of great value, however, unless the costs associated with this approach are clearly<br />

appropriate. Given that accessibility is a requirement, not just a nice thing to do, there are already significant<br />

costs to be borne by many content providers. The third approach lessens the need for all resources to be<br />

developed in a strictly universally accessible way just in case they are required. It also allows for the resources to<br />

be provided to individual users to be distributed, meaning that they can be shared amongst content providers.<br />

This is not new in itself, but it is new that this sharing can be dynamically applied. If a content provider has not<br />

made some content fully accessible, another can make available the necessary extra content and the system can<br />

combine the components, so the authoring of accessible content becomes more of a distributed, cumulative<br />

activity. Although there is no data to prove that this is more efficient and cheaper, there is reason to believe it<br />

will be and research to test this can take place as soon as there are sufficient metadata and systems available for<br />

such testing to be undertaken.<br />

Describing Control and Display Requirements for interoperability<br />

One aspect of interoperability is the ability to share the same kind of information with others using the same<br />

systems and acting with the same goals. Another is to work across devices including using different hardware<br />

and software without losing the necessary ‘look and feel’ that facilitates learner mobility between devices.<br />

W3C has a Device Independence Working Group, another focused on the Mobile Web and a third working on<br />

Evaluation and Repair. All three Working Groups produce specifications that are important to the<br />

interoperability of AccessForAll. These groups do not work specifically on specifications for learners but<br />

learners are included in the user groups of concern to them.<br />

The Device Independence Working Group is responsible for a protocol known as Composite Capabilities and<br />

Personal Preferences (W3C CC/PP). Its general aim is to enable the quick identification of the needs of a device,<br />

such as a mobile phone or a desktop computer, so that user requests will initiate delivery of resource components<br />

that are necessary for the correct functioning of the device. A typical example is that when a large display page<br />

such as a newspaper page is to be presented on a tiny phone screen, it needs to be accompanied by a<br />

transformation that will facilitate navigation through the content without relying on a global view of it. Another<br />

important example is given when the mode of transmission of the content has to change to accommodate the<br />

device or user. The Device Independence Working Group, like all other W3C groups, is directed to take into<br />

account the accessibility aspects of this, and so Personal Preferences are included. The resulting CC/PP protocol<br />

provides an interoperable way of automatically transmitting information to Web servers from devices. This is<br />

what AccessForAll wants to do and so it is important that AccessForAll is in harmony with the CC/PP protocol.<br />

The W3C Mobile Web Working Group wants to ensure that information about users is available for devices<br />

wherever they are but is concerned that such information is not available for abuse by other people or agents.<br />

This requirement is one that is shared by the AccessForAll: there is no need for a person to be identified with or<br />

by the set of needs and preferences they choose to use (Nevile, 2005).<br />

The W3C Evaluation and Reporting Working Group have developed a language for managing the evaluation of<br />

content, typically the validation of it against formal test criteria. It enables the evaluations to be both distributed<br />

and cumulative, and at least identified with their evaluator and the time and date of evaluation. AccessForAll<br />

descriptions of learners’ needs and preferences, and resources’ accessibility characteristics, need to be trusted<br />

and this information is therefore of interest in their case too.<br />

218


Describing Content Needs and Preferences and Resources for Exchange<br />

For a network delivery system to match individual learner needs with the appropriate configuration of a resource,<br />

two kinds of descriptions are required: a description of the individual learner’s needs and preferences and a<br />

description of the resource’s relevant characteristics. Experience has shown that by catering for needs, everyone<br />

can benefit, as has been the case with curb-cuts that are now used by people in wheel-chairs, children on skateboards<br />

and men with heavy trolleys.<br />

The AccessForAll approach involves specifications for describing learner needs and preferences that define a<br />

functional description of how a learner prefers to have information presented, how they wish to control any<br />

function in the application and what supplementary or alternative content they wish to have available. It was<br />

found that these were the three main classes of such needs that should be in the descriptions. Specifically:<br />

display requirements usually include the use of screen readers or enhancers, tiny phone displays, reading<br />

highlights, Braille, tactile displays, visual alerts and structural presentation;<br />

control requirements usually include the use of keyboard enhancements, onscreen keyboards, alternative<br />

keyboards, phone keypads, mouse emulation, alternative pointing, voice recognition and coded input, and<br />

content requirements usually include the use of alternatives to each of the modes of display (auditory, visual,<br />

tactile and what is classed as textual). It includes learner scaffolds, personal style sheets and extra time.<br />

AccessForAll deals with the needs and preferences of learners. It might seem unnecessary to be concerned about<br />

preferences at this level, but for some learners with disabilities, they have very limited means to access<br />

resources, and it may be essential to them that they have their exact needs satisfied. It is necessary to distinguish<br />

them from learners who have the capability to use other systems but prefer a particular set. It would be<br />

inappropriate to limit the acquisition of resources for the more flexible learners just because their first preference<br />

was not satisfied. There is also a third situation that needs to be considered; some learners can have dangerous<br />

conditions induced by certain sets of features, such as when flashing content causes them to have epileptic fits.<br />

(Many others are distracted by these and wish to avoid them, if possible.) For this reason, the three classes of<br />

essential, preferred, and prohibited need to be available to qualify the requirements.<br />

The AccessForAll approach requires finer than usual details with respect to embedded objects and for the<br />

replacement of objects within resources where the originals are not suitable on a case-by-case basis. Embedded<br />

objects include components such as images that appear integral to a Web page or resource. It is important that<br />

there is no significant difference between the handling of embedded and distributed components. Distributed<br />

components are those that form part of the Web page or resource but do not come from the same source as other<br />

components. Knowing which component is being rearranged in the AccessForAll process and having general<br />

information about the component will often be necessary when a system searches for an alternative to that<br />

component. In some cases, the creator of the appropriate alternative will not know of the existence of the original<br />

and certainly not have related the alternative to it in any way. This may mean that sometimes it will be necessary<br />

to determine the source of a component so that alternatives can be found by reference to it. For example, when<br />

looking for a video of signed language for use as an alternative to the sound track of a film of the play Hamlet, it<br />

may be possible to find a sign-language presentation of the play. Managing such a discovery challenge will<br />

benefit from interoperability.<br />

In defining requirements, AccessForAll does not mention the reason for any of the requirements. In some cases,<br />

learners with disabilities use assistive technologies to emulate other technologies, such as when a head-pointer is<br />

used to emulate the standard mouse so that as far as the functioning of the computer and the resource is<br />

concerned, there is no special accommodation. When the assistive technology impacts other technologies, as<br />

happens when a section of a screen is used for an on-screen keyboard, there are often detailed requirements not<br />

only for the display functions but also for the keyboard itself. Some learners need to specify the attributes of<br />

their keyboards, such as the size and separation of the keys, and others want to take advantage of features of the<br />

keyboard software they are using. Another kind of problem arises when a learner who has previously used a<br />

resource on a desktop computer tries to continue to use the resource on a telephone screen.<br />

The required interoperability happens at many levels.<br />

Interoperability between Needs and Preferences and Resources<br />

If learners are to be able to quickly configure their devices, they require their needs and preferences to be quickly<br />

recognized and implemented by the device they are using. Also, if they are to search for appropriate resources<br />

219


(including where their search for resources causes their system to search for accessible components from which<br />

to make the resource they want), their needs and preferences descriptions have to be available to the search<br />

engine for searching and matching with the resources and their components. Where this is happening across<br />

collections of resources, a common way of describing the resources will be necessary and it will need to mirror<br />

the descriptions of the resources. So interoperability between the two sets of descriptions is necessary so that,<br />

even though one is concerned with the user’s needs and the other with a resource, they can both be used by<br />

search engines. When a request is made for a resource, the search engine may need to look for both the user’s<br />

needs and preferences and the resource. Both will be necessary to constrain the search results to appropriate<br />

resources and their components. In effect, this means that for simplicity, the description of the learner’s needs<br />

should be in the same format as the descriptions of the resources.<br />

Interoperability between Devices<br />

Typically, learners with special needs will be looking for resource components that are developed by specialists.<br />

Usually, specialists who have not made the original resources produce closed captions, image descriptions and<br />

video files of people signing. They are likely to know the standard assistive technologies and what they will<br />

require and can do to use the special components. In automating the matching process for the learner, it is very<br />

important that the standard triggers are available for the assistive technologies. This means that the resources<br />

should be described in the way they can be understood by particular assistive technologies but also so that there<br />

is a generic description specification that all the assistive technologies can be expected to refer to. For this<br />

reason, care has been taken in AccessForAll to ensure that there is a seamless match and the established industry<br />

terms are used. The implications for interoperability here are for exchange between systems known as ‘user<br />

agents’, including browsers.<br />

It is well known that browser developers pride themselves on the non-standard features they offer and that it is<br />

not easy to satisfy all browser specifications simultaneously. Fortunately, assistive technology developers who<br />

have a much smaller market are often more concerned to serve their customers and their industry associations.<br />

Nevertheless, it is important to recognize their differences and allow for their use so the AccessForAll model has<br />

to be capable of such flexibility. In fact it aims for some generic functions to be described in a common way<br />

while allowing for extensions to accommodate custom functions or features.<br />

Interoperability across Information Sectors<br />

Many, but not all, resources used within educational institutions are described by or for the educational<br />

community, usually according to standards designed for the educational community. Having worked with the<br />

goal of sharing resources for some time now, the many educational communities have a number of ‘standards’,<br />

the best-known being the LOM developed by IEEE (the IEEE 1484.12.1-2002 Standard for Learning Object<br />

Metadata). Clearly, the accessibility characteristics of resources that are ‘learning objects’ need to be described<br />

in a way that interoperates with all other aspects of LOM descriptions.<br />

The IMS Global Learning Project, a consortium of educational organizations, has adopted the LOM and added<br />

the Learner Information Profile (LIP) that describes the attributes of the learner. AccessForAll was an IMS<br />

initiative and so it closely resembles their versions of the LOM and LIP in order to operate as an extension of<br />

them. In fact, the IMS versions are known as AccLIP and AccMD (IMS Global Learning Consortium, 2005).<br />

Often, however, educational activities involve learners using resources that have been developed and described<br />

by other communities for their own purposes. For example, technical manuals are often used in Computer<br />

Science courses but they are not usually written for this purpose. Government information is often used in<br />

education, as are images of paintings and objects held in museums and galleries. The resources to be used by<br />

learners then, do not always originate from the educational or even the same communities and their description<br />

for discovery purposes can be very specific to the community from where they come. In order to discover<br />

resources across communities, or disciplines then, the descriptions of the accessibility characteristics of<br />

resources need to be consistent with descriptions used in those communities.<br />

The simple Dublin Core Metadata Element Set (DCMES) is an ISO standard (ISO 15836) for core resource<br />

description metadata. There is also a set of qualified Dublin Core elements with additional terms and extensions<br />

(DC Metadata Terms). Dublin Core metadata is not domain specific. The Dublin Core Metadata Initiative<br />

(DCMI) has worked on cross-disciplinary metadata (DC metadata) for resource description and provides a base<br />

220


set of descriptive elements that support cross-domain discovery. Where domain specific communities want to<br />

extend the Dublin Core set, they develop what are known as Application Profiles for their community (DCMI,<br />

2003). Given that governments, museums and galleries, and other sectors use DC metadata for information<br />

sharing, AccessForAll aims to take advantage of the interoperability of DC descriptions. DC metadata also has<br />

the advantage that it is used in many countries for resources that are created in many different languages and so<br />

can be used for cross-language discovery.<br />

DC metadata is for describing resources and does not support rich descriptions of people but as people are not<br />

described in the learner needs and preferences profile, this is not a problem. It would be necessary, however, for<br />

there to be DC usable descriptions of learners’ needs and preferences but, as stated above, these need not be<br />

Dublin Core descriptions although, as argued elsewhere, they could be DC-style descriptions and described as<br />

DC resources to allow for their discovery (Nevile, 2005).<br />

Interoperability across Information Models<br />

DC Metadata and IEEE LOM and IMS LIP metadata belong to very different information models. DC metadata<br />

has a ‘flat’ structure with a set of ‘tags’ that can be applied individually or collectively to resources. IEEE LOM,<br />

and therefore IMS LIP, models are very different. IEEE LOM (and IMS LIP) structure is deeply hierarchical.<br />

The difficulty of matching information from ‘flat’ to ‘heirarchical’ models, and the reverse, has separated the<br />

two communities for some time. Recently there have been agreements to work towards a common structure in<br />

the future, and some attempts to find suitable temporary solutions for the present.<br />

The DC and IEEE LOM communities do share the use of standard syntax including HTML, XHTML,<br />

eXtensible Markup Language (XML), in particular, and the more narrowly defined Resource Description<br />

Framework XML (RDF). Both XML and RDF are extensible, however, so that they are used by two systems<br />

does not necessarily mean very much in terms of functional interoperability. On the other hand, it is this<br />

extensibility that makes it possible for the best attempts at interoperability.<br />

In summary, AccessForAll needs to interoperate with a number of other relevant metadata specifications and<br />

standards.<br />

The History of the AccessForAll Interoperability Efforts<br />

Originally, the AccessForAll descriptions were developed for an in-house application. The original prototype<br />

Inclusive Learning Exchange (TILE), was a system that chose among a set of resource components according to<br />

the learner’s needs and preferences. The resources were treated at the atomic level, each image and chunk of<br />

text, such as text articles, captions for other rich media resources, and so on, being separately stored and<br />

described. The learner’s needs and preferences information was set within the system and variable at any time<br />

the system was being used. The descriptive information was related in a hierarchical decision-tree that allowed<br />

for different levels of detail appropriate for the circumstances. The resource components in the learning system<br />

were described so that their potential for matching was available to the application that chose and assembled the<br />

components to be delivered as the resource. There was no need for interoperability with other systems in this<br />

case as the system was in-house and contained all the resources, components and user profiles.<br />

The original specifications for TILE were adopted by the IMS Global Learning Project with the aim that they<br />

would become interoperable by extending the IEEE LOM and the IMS LIP in a consistent way. They were<br />

dependent upon their common expression in an XML binding, inherited from TILE, for interoperability.<br />

The IMS Accessibility Special Interest Group sought permission to work towards an open standard with<br />

participation from others (including the DCMI Accessibility Working Group) in order to increase the<br />

opportunities for widespread adoption and interoperability. This collaboration led to the deconstruction of the<br />

original AccessForAll model and a new construction, without loss of particulars, as an abstract model (see<br />

below). During this process, it was crucial to the success of the outcome that it would not compromise its<br />

relationship with the LIP or the LOM.<br />

Before the adoption by DCMI of AccessForAll as a recommendation to the DC users community, and in<br />

anticipation of a revised, interoperable IMS version of the AccLIP and AccMD, the IMS specifications were<br />

forwarded to the International Standards Organisation (ISO JTC1 SC36) for adoption as a multi-part<br />

221


international standard. The ISO process involves the support and participation of a range of nations and helps<br />

foster international adoption of the specifications. According to the agreement by which the specifications were<br />

contributed by IMS, unlike many of the ISO standards, they will be free to those who want to use them.<br />

Enabling Interoperability with Abstract Models<br />

Between them, IEEE LOM and DC metadata describe many of the resources that are of interest in education.<br />

Some educational systems use LOM-based metadata sets to describe their resources but others use DC-based sets<br />

and others use a combination or different metadata. Although it is not clear how many do what, there has been<br />

careful analysis of the use of IEEE LOM and it is clearly an important player as is DC metadata, particularly as<br />

the latter is often used for government or document-type resources (Godby, 2004). It makes sense that these two<br />

communities should be able to exchange metadata records about their resources so they can, in fact, share their<br />

resources. To do this, they need to be able to transform metadata from one specification to the other. There is an<br />

activity, started in 2001 by the production of a joint article (Duval et al, 2002) that aims to bring the two sets of<br />

specifications into harmony. It cannot be done easily because LOM and DC metadata are based on very different<br />

information models.<br />

In 2004, to clarify the nature of DC metadata, the DCMI developed an abstract model of DC metadata (DCAM)<br />

that was adopted in early 2005. The model defines the scope and functions of DC metadata that enables its<br />

implementation using a range of syntaxes without loss. Where the syntax is standard, as where standard XML or<br />

RDF is used, both the structure (defined in the abstract model) and the syntax will be interoperable across all DC<br />

metadata applications. Such applications are formally known as application profiles (CWA14855).<br />

Once abstract models are agreed, it is relatively easy to develop information models (instantiations of the<br />

abstract model) and to create the syntax for representation of the information and then it is up to the community<br />

to agree on the terms to be used (semantics) for complete technical interoperability. The DCMI community<br />

developed their abstract model ten years after their information model, but fortunately for them, had a relatively<br />

simple information model. The IEEE LOM information model is extensive and complex and so far there is no<br />

corresponding abstract model for it.<br />

DCMI introduces its abstract model as follows:<br />

This document specifies an abstract model for DCMI metadata. The primary purpose of this<br />

document is to provide a reference model against which particular DC encoding guidelines can<br />

be compared. To function well, a reference model needs to be independent of any particular<br />

encoding syntax. Such a reference model allows us to gain a better understanding of the kinds of<br />

descriptions that we are trying to encode and facilitates the development of better mappings and<br />

translations between different syntaxes. (DCMI, 2005)<br />

In order to be recommended by the DC Usage Board for general adoption by the DC community, AccessForAll<br />

metadata needed to be explained in DC terminology. In effect, by late 2005, this meant it had to comply with the<br />

DC Abstract Model. As AccessForAll did not have an abstract model, developing one seemed like a good<br />

exercise. In the end, it was a very constructive exercise.<br />

The AccessForAll Abstract Model<br />

Developing abstract models is not easy when what they are to represent is already defined but in the case of<br />

AccessForAll, there was the opportunity to redefine the AccessForAll information model concurrently with<br />

developing its abstract model. The main problem was finding an abstract way to define AccessForAll that would<br />

maintain interoperability with all the relevant information models (structure), syntax (easier), semantics<br />

(relatively simple), and encourage systemic adoption.<br />

The AccessForAll Framework includes, significantly, both the user’s needs and preferences profile and the<br />

resource profile.<br />

In presenting its abstract model, DCMI provides a set of definitions and rules followed by a diagram expressed<br />

in a formal graphical representation format known as Unified Modeling Language (UML). The AccessForAll<br />

abstract model is now similarly described.<br />

222


To simplify the scope of the AccessForAll and to increase interoperability with other metadata, only those things<br />

that are necessary for the AccessForAll process are contained in its abstract model. Other descriptions should be<br />

managed according to other relevant metadata specifications.<br />

As unique definitions and rules, AccessForAll has the following:<br />

The abstract model of the preference sets defined in the AccessForAll framework is as follows:<br />

Each person has zero or more needs and preferences.<br />

Each needs and preferences has zero or more adaptability statements.<br />

Each needs and preferences has zero or more contexts.<br />

The abstract model of the resources defined in the AccessForAll framework is as follows:<br />

Each resource meets the needs and preferences (of a person).<br />

Each resource has zero or more adaptability statements.<br />

Each resource may be related to zero or more alternative resources.<br />

The abstract model of the adaptability statements defined in the AccessForAll framework is as follows:<br />

Each adaptability statement may contain access mode information.<br />

Each adaptability statement may contain zero or more related descriptions. [DC-AWG, 2005]<br />

Figure 1: A graphical representation of the AccessForAll abstract model.<br />

Figure 1 shows that a user, a person (a learner) or agent (perhaps a device or application), has needs and<br />

preferences that can be expressed as a resource, their Personal Needs and Preferences (PNP). This PNP is, or is<br />

closely associated with, an adaptability statement. It should describe the access mode they need or prefer. It may<br />

also have other descriptions (metadata) to make it easily discoverable and easily differentiated from other PNPs<br />

they use. Often a user will refer to the PNP they want simply by using a name for the context in which they use<br />

it. In some cases, the provision of needs and preferences information has already been tried on (Web-4-All)<br />

smart cards that can be used to provide some of this information.<br />

When the learner searches for a resource using the AccessForAll process, they want its adaptability statement, its<br />

Digital Resource Description (DRD), matched to their PNP adaptability statement. When a resource does not<br />

satisfy this rule, existing or alternative components or resources may need to be substituted, augmented or<br />

transformed until it does. As with the PNP, the resources they seek may have other descriptions but these,<br />

although useful, are not within the scope of the AccessForAll abstract model, except in as much as they may<br />

provide the context description.<br />

Adaptability statements can be very simple, merely showing a single attribute of the resource, or they can be<br />

quite complex, using the refinement of a refinement model to capture hierarchical information.<br />

Interoperability with DC Metadata<br />

The AccessForAll abstract model now closely matches the Dublin Core Abstract model. Not everything that will<br />

be useful to have as AccessForAll metadata is unique to the AccessForAll model so a significant amount of<br />

223


information will be expressed using standard DC elements. Exactly how to do this will be described in what is<br />

known as a DC Application Profile for which specific terminology (semantic values) will be defined. The value<br />

of this work for DC users is that they will be able to express the AccessForAll metadata in DC compliant ways<br />

so it will interoperate with other DC metadata. They will also be able to use standard DC applications without<br />

significant modification.<br />

Interoperability with IEEE LOM and IMS LIP Metadata<br />

An encoding of AccessForAll metadata for use in an IEEE LOM Application is under construction by the CEN-<br />

ISSS Learning Technologies Workshop (CEN/ISSS (WS-LT), Accessibility Properties for Learning Resources<br />

Group.<br />

Nilsson et al. (2005) have recently worked on developing what they call a future metadata standardization<br />

framework. They say:<br />

We have demonstrated that true metadata interoperability is still, to a large extent, only a vision,<br />

and that metadata standards still live in relative isolation from each other. The modularity<br />

envisioned in application profiles is severely hampered by the differences in abstract models<br />

used by the different standards, and efforts to produce vocabularies often end up in the dead end<br />

of a single framework. In order to enable automated processing of metadata, including<br />

extensions and application profiles, the metadata will need to conform to formal metadata<br />

semantics.<br />

To achieve this, there is a need for a radical restructuring of metadata standards, modularization<br />

of metadata vocabularies, and formalization of abstract frameworks. RDF and the Semantic Web<br />

provide an inspiringly fresh approach to metadata modeling: it remains to be seen whether that<br />

framework will be reusable for learning object metadata standards.<br />

This suggests that it may not be until there is a shared LOM/DC abstract model for education that there will be<br />

perfect interoperability between DC and LOM resource descriptions but it is hoped to be possible sooner, in the<br />

particular case of AccessForAll metadata, because it is based on a more interoperable abstract model.<br />

Adoption of specifications and standards<br />

The value of specifications and standards is only known some time after their release, when it becomes possible<br />

to gauge how widely they have been adopted and how well they have solved identified problems for<br />

organizations. Insurance that they will be adopted is not possible but open development of specifications,<br />

consensus among a wide range of types of implementers, and points of contact that support dissemination of the<br />

specifications are all known to help in the process.<br />

In the case of AccessForAll, the needs and preferences have come from those who actually have them; people<br />

who use technology in a wide range of situations, overcoming what often seem like insurmountable odds.<br />

The needs and preferences are not new, they have been around since computers were first developed and are<br />

tried and tested by their users. The provision of statements that describe needs and preferences is new. It is to be<br />

done in a way that separates the user from the needs and preferences, while allowing them to have several sets.<br />

Nevertheless, there is little doubt that this will not be a problem for many as already these settings have to be<br />

entered into computer systems. In addition, there is more pressure on those providing education and training to<br />

be more mindful of the special needs of their clientele, so they will be alerted to the need for such specifications<br />

in the normal course of business, as well as by those specifically promoting these specifications. The only<br />

difference for those already registering such needs is that they will be entered once and used many times. The<br />

experience of several projects where such specifications have been stored on smart cards for re-use is wellknown<br />

in the field and has contributed to this work (Web-4-All).<br />

The description of the accessibility characteristics of resources is new but it has been foreshadowed by the<br />

development of applications that help organizations assess the accessibility of their resources, and to record their<br />

characteristics in metadata. The difference is that rather than just do this for evaluation and penalty avoidance,<br />

224


organizations can do this now to promote accessibility by preparing their collection of resources to interoperate<br />

with others.<br />

The last community that needs to be engaged with the specifications is the industry that develops the<br />

technologies that are used for accessing resources. These players have been consulted and provide more or less<br />

of the relevant features. They have indicated that what is proposed will work for them.<br />

Finally, there is the question of who will use metadata anyway? If, as hoped, the distribution and accumulation<br />

of accessibility solutions is enabled by the specifications, and those responsible for providing learning resources<br />

are increasingly doing this through systems that work on metadata, it can only be hoped that they will be<br />

adopted. Presentations, workshops, journal articles, and other forms of dissemination are already active but in the<br />

end it will probably be good references from one person to another that will provide the final evaluation.<br />

Future Work and Conclusions<br />

In summary, the interoperability of the AccessForAll strategy is relevant in many circumstances. The process of<br />

making AccessForAll has involved significant technical work and collaboration between representatives of<br />

several communities, all with the goal of making education more accessible. The associated specifications show<br />

how the AccessForAll strategy can be implemented. They are not prescriptive about the encoding that should be<br />

used. Significantly, they are not prescriptive about what constitutes accessibility. The work that follows to<br />

support the adoption and implementation of the AccessForAll approach will be the true test of its<br />

interoperability: nothing is seriously interoperable unless people use it to interoperate. In addition, the network<br />

effects, upon which ultimate success will depend, cannot be expected to show until there is widespread adoption<br />

of the approach.<br />

There are endless opportunities, given the model and strategy, to take further advantage of new technologies.<br />

The Semantic Web offers one obvious technology that will be enabled by the AccessForAll approach. Already<br />

the first AccessForAll specifications recommend using the Semantic Web Evaluation and Reporting Language<br />

(EARL) so that the metadata will be as flexible and rich as possible. The range of other extensions includes<br />

opportunities for valuable cross-lingual exchanges to suit learner needs as well as cross-disciplinary changes of<br />

emphasis. Applications and Web services that transform resources or resource components to suit the needs of<br />

users with cognitive disabilities is a huge area that has hitherto not received the attention it deserves. The<br />

interoperability of specifications that purport to implement the new AccessForAll is clearly a major factor that<br />

will determine its effectiveness.<br />

Acknowledgements<br />

The authors wish to contribute to the valuable work being done by others and welcome involvement in their own<br />

work. They specially wish to acknowledge the contributions of collaborators Anastasia Cheetham, Martyn<br />

Cooper, Andy Heath, Madeleine Rothberg and David Weinkauf.<br />

Glossary<br />

AccessForAll: an Accessibility Framework: http://dublincore.org/accessibilitywiki/AccessForAllFramework.<br />

CWA14855, CEN Workshop Agreement, Dublin Core Application Profile guidelines:<br />

http://www.cenorm.be/cenorm/businessdomains/businessdomains/isss/cwa/cwa14855.asp.<br />

CEN/ISSS (MMI-DC), European Committee for Standardization, Meta-Data (Dublin Core) Workshop:<br />

http://www.cenorm.be/cenorm/businessdomains/businessdomains/isss/activity/wsmmi.asp.<br />

CEN/ISSS (WS-LT), Workshop on Learning Technologies:<br />

http://www.cenorm.be/cenorm/businessdomains/businessdomains/isss/activity/wslt.asp.<br />

DCAM, Dublin Core Metadata Initiative Abstract Model: http://dublincore.org/documents/abstract-model/.<br />

DCMES, Dublin Core Metadata Element Set: http://dublincore.org/documents/dces/.<br />

225


DC Metadata Terms: http://dublincore.org/documents/dcmi-terms/.<br />

IEEE 14.84.12.1 - 2002 Standard for Learning Object Metadata: http://ieeeltsc.org/.<br />

IMS Global Learning Consortium: http://www.imsglobal.org/accessibility/.<br />

ISO JTC1 SC36: http://jtc1sc36.org/.<br />

NLS: http://lcweb.loc.gov/nls/.<br />

Object Management Group, The Unified Modeling Language: http://www.uml.org/.<br />

STSN: http://www.stsn.org/servicechart.html.<br />

TILE, The Inclusive Learning Exchange: http://www.inclusivelearning.ca/.<br />

UsableNet Lift text Transcoder:<br />

http://www.usablenet.com/products_services/text_transcoder/text_transcoder.html.<br />

W3C CC/PP, Composite Capabilities and Personal Preferences: http://www.w3.org/Mobile/CCPP/.<br />

W3C EARL, Evaluation and Reporting Language: http://www.w3.org/TR/EARL10/.<br />

W3C HTML, HyperText Markup Language: http://www.w3.org/TR/html4.<br />

W3C RDF, Resource Description Framework: http://www.w3.org/RDF/.<br />

W3C WAI, W3C Web Accessibility Initiative: http://www.w3.org/WAI/.<br />

W3C WAI, Web Content Accessibility Guidelines for creating accessible Web pages:<br />

http://www.w3.og/TR/WAI-WEBCONTENT/.<br />

W3C/WAI WCAG 2.0: http://www.w3.org/TR/WCAG20/.<br />

W3C XHTML, Module-based XHTML: http://www.w3.org/TR/xhtml1.<br />

W3C XML, Extensible Markup Language: http://www.w3.org/XML/.<br />

Web-4-All: http://web4all.atrc.utoronto.ca/.<br />

References<br />

Chapman, A. (200). Library services for visually impaired people: a manual of best practice, Chapter 10,<br />

retrieved 24 August <strong>2006</strong> from http://bpm.nlb-online.org/contents.html.<br />

DC-AWG (2005). Dublin Core Accessibility Working Group, IMS Accessibility Working Group, ISO JTC1<br />

SC36 WG7, AccessForAll: an Accessibility Framework retrieved 24 August <strong>2006</strong> from<br />

http://dublincore.org/accessibilitywiki/AccessForAllFramework.<br />

DCMI, 2003. DCMI Usage Board Review of Application Profiles retrieved 24 August <strong>2006</strong> from<br />

http://dublincore.org/usage/documents/profiles/index.shtml.<br />

DCMI (2005). Dublin Core Metadata Initiative Abstract Model, retrieved <strong>October</strong> 7, <strong>2006</strong> from<br />

http://dublincore.org/documents/abstract-model/.<br />

Disability Rights Commission (UK), (2004). The Web: Access and Inclusion for Disabled People, retrieved<br />

<strong>October</strong> 7, <strong>2006</strong> from http://www.drc-gb.org/publicationsandreports/report.asp.<br />

226


Duval, E, Hodgins, W. Sutton, S., & Weibel, S. L. (2002). Metadata Principles and Practicalities, in D-Lib<br />

Magazine, 8 (4), retrieved 24 August <strong>2006</strong> from http://www.dlib.org/dlib/april02/weibel/04weibel.html.<br />

Godby, J. (2004). What Do Application Profiles Reveal about the Learning Object Metadata Standard? retrieved<br />

24 August <strong>2006</strong> from http://www.ariadne.ac.uk/issue41/godby/.<br />

IMS Global Learning Consortium (2005). Accessibility for Learner Information Profile, and Accessibility Metadata,<br />

retrieved 24 August <strong>2006</strong> from http://www.imsglobal.org/accessibility/.<br />

Nevile, L, (2005). Anonymous Dublin Core Profiles for Accessible User Relationships with Resources and<br />

Services, Proceedings of the International Conference on Dublin Core and Metadata Applications, 67-78.<br />

Madrid: Universidad Carlos III de Madrid.<br />

Nilsson, M., Johnston, P., Naeve, A., & Powell, A. (2005). The Future of Learning Object Metadata<br />

Interoperability. In Koohang, A. (Ed.), Principles and Practices of the Effective Use of Learning Objects,<br />

Informing Science Press.<br />

227


Choquet, C., & Corbière, A. (<strong>2006</strong>). Reengineering Framework for Systems in Education. Educational Technology &<br />

Society, 9 (4), 228-241.<br />

Reengineering Framework for Systems in Education<br />

Christophe Choquet and Alain Corbière<br />

LIUM Laboratory, University of Maine, IUT of Laval, 53020 Laval Cedex 9, France<br />

christophe.choquet@lium.univ-lemans.fr<br />

alain.corbiere@lium.univ-lemans.fr<br />

ABSTRACT<br />

Specifications recently proposed as standards in the domain of Technology Enhanced Learning (TEL),<br />

question the designers of TEL systems on how to put them into practice. Recent studies in Model Driven<br />

Engineering have highlighted the need for a framework which could formalize the use of these<br />

specifications as well as enhance the quality of the developments. This paper deals with the opportunity for<br />

the TEL community to adopt such a model to express and formalize the design organization and the<br />

engineering process of a TEL system. This kind of model could provide a set of concepts allowing the<br />

description of specifying, modeling and analyzing tasks one needs to perform for defining the negotiation<br />

and the communication between actors in such a community. In a first part, we stress the need for the<br />

instantiation of this framework so as to integrate recent results as well as taking into account the evolution<br />

of software reengineering. In a second part, we propose two instances of this framework; the first concerns<br />

the reverse engineering of a TEL system and the second its reengineering.<br />

Keywords<br />

TEL engineering, TEL reengineering, TEL development process, Model driven engineering<br />

Introduction<br />

The design of TEL (Technology Enhanced Learning) systems calls for the communication and sharing of<br />

information about the users of these systems. In recent studies, the standards and specifications proposed by the<br />

TEL community have provided interoperable languages for the specification of the learning session’s<br />

progression, in order to enhance the communication between the actors of the development. These proposals are<br />

mainly focused on the a priori specification of the system’s behavior. But (Wenger, 1987; Bruillard & Vivet,<br />

1994; Hummel et al., 2004; Tchounikine et al., 2004) for instance, have shown that the design methodology of a<br />

TEL system cannot be reduced to the specification but must also instrument the tasks of recording and analyzing<br />

the uses of the system. Our objective is to merge TEL community results on standards and specifications, and<br />

proposals made by the software engineering community. In this way, we aim to support the communication<br />

between actors of development in a general reengineering framework, as Chikofsky (1990) for who<br />

"reengineering, also known as both renovation and reclamation, is the examination and alteration of a subject<br />

system to reconstitute it in a new form and the subsequent implementation of the new form".<br />

Recent proposals of TEL standards and specifications lead to the adoption of new design practices and<br />

methodologies, but they don’t provide designers with a set of concepts allowing the explicitation of the<br />

organization and the implemented engineering. Some development solutions are provided by the Object Oriented<br />

Software Engineering community, especially in the domain of software reengineering. With an empiric approach<br />

first, the open source and free software communities have had to develop specific and effective tools which<br />

support the structured communication around a design artifact. At the same time, the research community on<br />

software engineering has provided methodological developments as well as a standardized and shared<br />

vocabulary for software reengineering focused on the interoperability and reusability points of view. Chikofsky<br />

(1990) for instance, proposed a first terminology dedicated to the definition of the tasks of reverse engineering<br />

and reengineering. More recently, and based on this taxonomy, the European project ESPRIT 21975 (Bär et al.,<br />

1999) has provided a set of patterns and pattern languages which define and standardize both methodological<br />

concepts – such as “capture model details” or “negotiate a design” – and vocabulary – such as “process” or<br />

“structure”.<br />

The aim of this paper is to show how these results from the software engineering community could be used to<br />

define a general framework, able to support a set of designers, considered as a community, involved in the<br />

reengineering process of a TEL system. We propose to make an instance of the RM-ODP (Reference Model of<br />

Open Distributed Processing) framework on the reengineering process of a TEL system. This reference model is<br />

considered as a framework which coordinates and standardizes open distributed processes by the fine definition<br />

of a set of concepts on engineering practices. Aims and motivations of this RM-ODP standard are detailed in the<br />

document ISO/IEC 10746-1 (ISO/IEC 10746-1, 1998).<br />

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the<br />

copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies<br />

are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by<br />

others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior<br />

specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.<br />

228


In a first part, we present the instantiation of the standard RM-ODP on the reengineering process of a TEL<br />

system. Then, we provide two use cases of the model we have proposed. The first one highlights a reverse<br />

engineering task where both an instructional designer and a software engineer are involved. The second one<br />

shows how, during the reengineering of a TEL system, the communication between an instructional designer and<br />

an analyst specialized in the interpretation of users logs could be supported.<br />

A General Framework for the Reengineering of a TEL System<br />

Reference Model of Open Distributed Processing Presentation<br />

RM-ODP is defined by its authors as a generic framework aimed to support the modeling process of a complex<br />

and distributed system. For this purpose, it stresses the need for the designers to refine within their own domain,<br />

a set of generic concepts, such as object composition/decomposition, object state and behavior, viewpoints…<br />

These concepts are declined following three modeling phases introduced by the model:<br />

the system specification, where designers classify and make up the different objects of the system;<br />

the system modeling which defines, for different levels of abstraction, the models of interaction between<br />

objects;<br />

the system structuring, where the different structures which should be implemented in the system are<br />

defined.<br />

The aim assumed by RM-ODP is to help a community of designers to adopt a consensus (e.g. a standard)<br />

regarding the designing process of their system. With this in mind, RM-ODP is structured into three<br />

complementary documents. The first one, the “Overview” (ISO/IEC 10746-1, 1998) defines the model’s<br />

application spectrum. The second one, “Foundations” (ISO/IEC 10746-2, 1996), defines the set of generic<br />

concepts related to the specification, modeling and structuring phases. The last one, “Architecture” (ISO/IEC<br />

10746-3, 1996) is dedicated to the definition of five viewpoints on the design of a system architecture, as well as<br />

the definition of the different languages associated with these viewpoints. These are:<br />

The enterprise viewpoint: A viewpoint on the system and its environment that focuses on the purpose, scope<br />

and policies for the system, represented with concepts such as community, actor, role, activity and tool.<br />

The information viewpoint: A viewpoint on the system and its environment that focuses on the semantics of<br />

the information and information processing performed.<br />

The computational viewpoint, which is concerned with the functional decomposition of the system into a set<br />

of objects that interact at interfaces – enabling system distribution.<br />

The engineering viewpoint: A viewpoint on the system and its environment that focuses on the mechanisms<br />

and functions required to support distributed interaction between objects in the system.<br />

The technology viewpoint: A viewpoint on the system and its environment that focuses on the choice of<br />

technology in that system.<br />

Specifying an architecture in the RM-ODP framework allows designers to focus their attention with these five<br />

viewpoints (see Figure 1). They are the main proposals of RM-ODP which allow the mastering of a complex<br />

system specification..<br />

Metamodel : Foundation<br />

information viewpoint<br />

Enterprise viewpoint<br />

engineering viewpoint<br />

technology viewpoint<br />

computational viewpoint<br />

Figure 1. The five viewpoints of RM-ODP<br />

Specification and<br />

Modeling acts<br />

Structuring acts<br />

229


TEL Standards, Specifications and Projects: Part of the Instantiation of RM-ODP<br />

Recent works and projects on TEL engineering, conducted by several standardization committees, consortiums<br />

and scientific and industrial communities, want to define formalisms, engineering methods and/or platforms<br />

which focus on a specific aspect of a TEL system, whilst remaining neutral from a pedagogical point of view.<br />

These approaches are close to the concept of viewpoint as defined by the RM-ODP standard, which allows the<br />

specification of different aspects of a system within a same framework.<br />

The information viewpoint. The LOM (Learning Object Metadata) standard of the IEEE (Institute of Electrical<br />

and Electronics Engineers) (IEEE 1484.12.1, 2002) is a meta-descriptor for a learning object which allows<br />

indexation in a distributed databank for reuse. The EMLs (Educational Modeling Languages) (Rawlings et al.,<br />

2002), and especially the informational models proposed by the IMS Consortium, like Learning Design (IMS-<br />

LD/IM, 2003), are languages which allow the definition of an instructional scenario by the association between<br />

informational concepts such as roles or activities. The related constraints are defined by informational models in<br />

several technical documents. These two main proposals could be considered as the instantiation of the<br />

information viewpoint of RM-ODP on the TEL engineering: they allow a coherent description of both the<br />

learning objects and the scenarios of a TEL system. The proposed formalisms bring designers to use a formal<br />

syntax for their descriptions.<br />

The computational viewpoint. One of the three documents provided by IMS for its Learning Design<br />

Specification, named "Best Practice Implementation Guide" (IMS-LD/BP, 2003), describes the functional<br />

decomposition of a TEL system through the specification of links, interfaces and interactions between objects.<br />

Earlier, the LTSA (Learning Technology Systems Architecture) proposal (IEEE- P1484.1/D9, 2001) of IEEE<br />

aimed to define an architectural model to highlight the functional components and the different flows existing in<br />

a learning process. These two proposals could be considered as the instantiation of the computational viewpoint<br />

of RM-ODP on the TEL engineering.<br />

The engineering viewpoint. SCORM (Sharable Content Object Reference Model) (ADL, 2004) proposed by<br />

ADL (Advanced Distributed Learning) and "Abstract Framework" (IMS-AF/WP, 2003) provided by IMS, could<br />

be considered as the instantiation of the engineering viewpoint of RM-ODP on the TEL engineering. These two<br />

proposals synthesize the specification of the technical framework of a TEL system. Freeware community<br />

projects such as OpenUSS (2005) or industrial specifications such as E-Learning Framework (SUN, 2003) are<br />

relevant of the consensus adopted in the IMS document. These proposals stress the need for the designers to<br />

know and take into account the characteristics of the interfaces and behaviors of the different components of a<br />

TEL system.<br />

The technology viewpoint. We don't aim to elaborate here on the necessity for the TEL community to adopt<br />

software approaches as Model Driven Architecture proposed by OMG (2001), but we think that the different<br />

TEL projects which are relevant to this approach, like OpenUSS (Grob et al., 2004), FreeStyle Learning (Brocke,<br />

2001), or Future Learning Environment (Leinonen et al., 2002) are good examples of the instantiation of the<br />

technology viewpoint of RM-ODP in a TEL project.<br />

This short panorama of the TEL community results allows us to focus on the shortcomings of these works. The<br />

initial and main objective of these projects, especially the standards proposals, is to enhance the interoperability<br />

between TEL systems and the reusability of their productions. As a consequence, all of these propositions lead to<br />

the instrumentation the specification task, neglecting the iterative dimension of the development of an<br />

educational system. The instantiation of RM-ODP on TEL engineering highlights the weakness of the enterprise<br />

viewpoint in our TEL community. In particular, we want to underpin a meta reflection on the life cycle of a TEL<br />

system, on languages used in each viewpoint, and on interrelations existing between all actors of the iterative<br />

process of a TEL development. This is the reason we propose below an enterprise viewpoint for the<br />

reengineering of a TEL system.<br />

The Enterprise viewpoint for reengineering of a TEL System<br />

The quality approach of RM-ODP concerns the organization of the actors, the documents and the phases of the<br />

system life cycle. Quality could be defined here by the quantity of observed uses which have been taken into<br />

account in a reengineering purpose, but also by the research of qualitative missing indicators (ISO/IEC 10746-1,<br />

1998, Chapter 9.1). The enterprise viewpoint of RM-ODP allows the explicitation of the organizational<br />

dimensions related to such an iterative process in a unique framework.<br />

230


Pawlowski (2002) is a first approach for identifying and taking into account the specific enterprise aspects of a<br />

TEL development. But it is restricted and only synthesizes the different projects in educational systems quality<br />

insurance. Moreover, this work does not contribute to defining a unique framework including other approaches<br />

such as, for instance, the existing standard proposals. Oubahssi et al. (2004) want to define a new meta descriptor<br />

which allows the designers to describe each type of learning object (which here could be a web document, a<br />

video,… but also a scenario, a service, a role, …) during all phases of the design, the use and the analysis of a<br />

TEL system. But the enterprise viewpoint is not supported here: only the computational and information<br />

viewpoints are taken into account.<br />

The proposal of an enterprise viewpoint for the reengineering of TEL systems we make here, as shown in Figure<br />

2, aims to integrate both the different constraints existing in a reengineering process and the specificities of the<br />

use of TEL standards. We use the vocabulary described in (ISO/IEC-15414, 2002):<br />

Process: a collection of steps taking place in a prescribed manner and leading to an objective. A Process is a<br />

set of tasks which operate and produce information for reaching and operational objective.<br />

Community: a collection of enterprise objects which share the same objectives. A Community is constituted<br />

by humans, informative resources and/or tools which share the same objective into the system under<br />

specification.<br />

Enterprise Object: a basic entity of the enterprise specification to fulfill one or more roles. An Enterprise<br />

Object could be any component of the system specification which could assume at least one basic function<br />

in the operational system.<br />

Role: abstraction of an enterprise object behavior or a community behavior. A Role emphasizes the<br />

comportment of one entity or a set of them.<br />

Actor: an enterprise object that participates in the action. Thus, the term Actor qualifies the entities which<br />

participate to a named interaction with other objects.<br />

Action: Action is a basic modeling concept to qualify each object within an ODP system. An Action allows<br />

the definition of an interaction inside an object or between several objects.<br />

Objective: a practical advantage or intended effect, expressed as preferences about future states.<br />

Behavior: A behavior of an object is a collection of actions that the object may take part in, together with<br />

the set of constraints on when those actions can occur.<br />

Artifact: an enterprise object that is referenced in the action, that is to say, that is necessary for performing<br />

the action.<br />

Resource: an enterprise object which is essential to some behavior.<br />

Step: An abstraction of an action, used in a process that may leave unspecified objects that participate in that<br />

action. This term allows the definition of actions prototypes.<br />

Graphical conventions are taken from the ECA (Enterprise Collaboration Architecture) UML profile, as<br />

proposed by the OMG consortium (OMG/ECA, 2004). This profile aims to support a model driven development<br />

process, in an RM-ODP framework (Nagase et al., 2004).<br />

The enterprise viewpoint we propose for the TEL systems reengineering is composed of six processes, described<br />

here with the help of the vocabulary and rules proposed by RM-ODP.<br />

The Design Process produces specifications guided by the main standardized informational models of TEL (e.g.<br />

LOM and Learning Design). These specifications need to be adapted to existing resources (learning objects) and<br />

learner profiles, and/or modified, taking into account information produced by the Analysis Process (uses<br />

analysis, system comportment).<br />

The Software Engineering Process interprets the designers' specification and produces a prototype of the TEL<br />

system (we consider here each system as a prototype which could be modified by reengineering). This process<br />

has to negotiate with the Design Process and the Analysis Process in order to define the observation needs of a<br />

learning session and to develop and integrate the related software captors.<br />

The Learning Process supports the learning sessions. It reifies the scenario specified by the Design Process and<br />

implemented by the Software Engineering Process in the prototype. This scenario leads to the organization of<br />

some actions, named activities in Learning Design, performed by the actors' roles of the learning session.<br />

The Analysis Process operates computational and manual use analysis techniques. Its negotiation with the<br />

Software Engineering Process could establish new needs: events to observe and/or new interpretations. Uses<br />

analysis feedback is provided to the Design Process.<br />

231


The Resources Management Process manages the learning objects resources.<br />

The Learners Profiles Management Process manages the enterprises objects, named here audits, of the Analysis<br />

and Design Processes of the TEL system reengineering community. These objects are mainly structured<br />

knowledge, provided as, for example, described scenarios in the meaning of (Iksal & Choquet, 2005; Barré &<br />

Choquet, 2005).<br />

The processes take part in the actions of the TEL system reengineering process community. The definition of a<br />

community (see above) allows us to consider each process itself as a community of enterprise objects. We will<br />

describe the objectives, roles and processes of these communities with two examples, one on the reverse<br />

engineering of a TEL system, the other on its reengineering.<br />

Use Cases<br />

TEL System Reengineering<br />

Community<br />

Software<br />

Engineering<br />

Process<br />

use a<br />

resource<br />

produce a<br />

prototype<br />

adapt a<br />

component<br />

produce a<br />

model<br />

interpret a<br />

specification<br />

Resources<br />

Management Process<br />

provide a<br />

resource<br />

store a<br />

resource<br />

provide a<br />

resource<br />

analyze a<br />

query<br />

Figure 2: UML package of the enterprise viewpoint of the reengineering process of a TEL system<br />

Presentation of the TEL system<br />

Learning<br />

Process<br />

execute a<br />

prototype<br />

Design<br />

Process<br />

produce a<br />

specification<br />

produce a<br />

resource<br />

use a<br />

resource<br />

search a<br />

resource<br />

produce an<br />

observable<br />

interpret a<br />

model<br />

interpret an<br />

information<br />

interpret an<br />

audit<br />

Analysis<br />

Process<br />

analyze an<br />

observable<br />

ask for an<br />

adaptation<br />

produce an<br />

information<br />

make an<br />

interpretation<br />

of an audit<br />

Learners Profiles<br />

Management Process<br />

produce an<br />

audit<br />

ask for an<br />

audit<br />

produce an<br />

audit<br />

manage an<br />

audit<br />

For four years, we have been testing a TEL system with a reengineering approach using higher degree students<br />

of a Computer Technology Institute and their teachers. Each year around fifty students participate to a two-hour<br />

learning session. After each session, teachers – who are the designers – and ourselves – who play the roles of<br />

software engineers and analysts, modify the system according to the uses analysis results. We will exemplify the<br />

use cases with this test as described briefly below.<br />

During a learning session, the students learn the main principles of the operationalization of a HTTP server with<br />

the deployed educational environment. The session is a part of a well identified course within the cursus<br />

provided by the Institute. The initial instructional scenario defined by designers was organized around 6<br />

activities supported by, at least, one web resource.<br />

The activity "Introducing the Learning Unit" is supported by a video which presents the main characteristics<br />

of a web server. This activity aims to present both the background knowledge mobilized by the learning<br />

unit, and the set of know-how the learner will acquire.<br />

232


The activity "Going into the Matter" is supported by a HTML document, composed of texts and figures<br />

which describe the main web protocols a web server has to support, and the programming tools needed to<br />

operate them.<br />

The activity "Linking Concepts and Methods" is supported by two slide shows, one presenting the relevant<br />

concepts, and the other describing, step by step, the execution of the Java code of a server.<br />

The activity "Training to Understand" is supported by several exercises on the HTTP server programming.<br />

The activity "Learning by Doing" is supported by a simulation where the learner, helped by a Java editor<br />

and a compiler, should modify existing code in order to make a functional web server.<br />

The activity "Evaluating the acquisition" is supported by two multiple choice questionnaires.<br />

As we have defined before, the enterprise viewpoint of a TEL system is composed by six processes (design,<br />

software engineering, learning, analysis, resources management and learners profiles management). Each of<br />

them should use software such as the open software projects – referenced in the Table 1 – as we have used for<br />

our test.<br />

Process<br />

Table 1: Open Software Projects Used in the Test<br />

Open software products<br />

Design Process Mozilla (Mozilla, 2005), OpenOffice (OpenOffice, 2005), OpenUSS<br />

(OpenUSS, 2005), FreeStyle Learning (FSL, 2005)<br />

Software Engineering Process Eclipse (Eclipse, 2005), AndroMDA (AndroMDA, 2005),<br />

PoseidonCE (PoseidonCE, 2005)<br />

Learning Process Mozilla, OpenUSS, FreeStyle Learning<br />

Analysis Process Weka (Weka, 2005), WUM (WUM, 2005)<br />

Resources Management Process GForge (GForge, 2005), CVS (CVS, 2005)<br />

Learners Profiles Management Process OpendLDAP (OpenLDAP, 2005)<br />

The Use Case of a Reverse Engineering of an Educational System<br />

On page 15 of Chikofsky (1990), the reverse engineering is defined as "the process of analyzing a subject system<br />

with two goals in mind: 1) to identify the system's components and their interrelationships; and, 2) to create<br />

representations of the system in another form or at a higher level of abstraction". We will here present a typical<br />

use case of reverse engineering which explains how the RM-ODP framework could guide the software<br />

engineering process when it tries to implement the specification (a representation of the TEL system from the<br />

information viewpoint) provided the design process. The interaction between these two processes produces an<br />

artifact which is a representation of the TEL system from the technology viewpoint. This use case is structured in<br />

five steps, as shown in figure 3.<br />

Foundations<br />

<br />

Information viewpoint<br />

<br />

<br />

Enterprise viewpoint<br />

Engineering viewpoint<br />

Technology viewpoint<br />

Computational viewpoint<br />

<br />

<br />

Figure 3: The five steps of the Reverse Engineering Use Case<br />

233


Step 1: Instantiation of the enterprise viewpoint. The reverse engineering of a TEL system needs a dialog<br />

between the software engineering process and the design process. It is this interaction between these two<br />

enterprise objects, grouped in a same community, which is modeled from the enterprise viewpoint, as shown in<br />

Figure 4 with ECA notation.<br />

Reverse Engineering<br />

Community<br />

Design<br />

Process<br />

t<br />

Designer / Engineer<br />

Protocol<br />

Engineering<br />

Process<br />

Engineer / Designer<br />

Protocol<br />

Figure 4: Instantiation of the Enterprise Viewpoint<br />

Step 2: Instantiation of the information viewpoint. The role of the design process, instigator of the dialog, is to<br />

produce a specification of the TEL system, from the information viewpoint. We have used the different<br />

information formalisms provided by the IMS consortium for our test. Figure 5 is an excerpt from the<br />

instructional scenario, modeled in IMS Learning Design and IMS Content Packaging. It shows the environment,<br />

made up of two objects, chosen by the designers to support the activity "Going Into Matter". The object<br />

"knowledge-object" is the HTML document "author_text1.html" which contains the pedagogical content. The<br />

object "tool-object" is the Java component "manager.jar" which handles the diffusion of the document. In fact,<br />

according to Koper (2001), when designers choose to present the pedagogical content ("knowledge-object"),<br />

they explicit the need of the support object ("tool-object").<br />

<br />

<br />

<br />

<br />

<br />

<br />

<br />

<br />

…<br />

<br />

<br />

<br />

…<br />

<br />

<br />

<br />

Figure 5: Excerpt of the Instructional Scenario<br />

Step 3: Instantiation of the software engineering viewpoint. When the design process provides its specification,<br />

the software engineering process needs to identify and characterize, in a modeling task, the states of the objects<br />

introduced by the specification. Guided by the RM-ODP framework, a software engineer knows that he needs to<br />

identify three schemas (ISO/IEC 10746-3, 1996, Chapter 6.1):<br />

the Invariant Schema is a set of predicates and relations on information objects which are always true;<br />

the Static Schema defines the state of the information objects at some point in time;<br />

the Dynamic Schema is a specification of the allowable state changes of the information objects, subject to<br />

the constraints of an invariant schema.<br />

Here, the static schema is the set of models defined by the proposals Learning Design and Content Packaging of<br />

IMS and Content Package of ADLNet, to which are issued the information objects of the designers' specification<br />

(imscp:file, imscp:resource, imsld:item, imsld:environment, imsld:learning-object and adlcp:scormtype). The<br />

t<br />

234


invariant schema defines the existing links between these different objects, as it is shown in the Figure 6, with<br />

the help of the UML graphic language for a class diagram (OMG/UML, 2003).<br />

imsld:environment imsld:learning-object imscp:resource<br />

2<br />

Figure 6: Invariant Schema<br />

The dynamic schema does not exist at this state of the design. Thus, the software engineering process needs to<br />

identify and model new information objects, in order to establish the dynamic schema. One needs here to<br />

characterize the behavior of these different objects.<br />

Step 4: Instantiation of the computational viewpoint. To characterize the dynamic schema, the software<br />

engineering process needs more information on the specification provided by designers. The pattern "Capturing<br />

model details" proposed by Demeyer et al. (2002), supports such a reverse engineering approach. From the<br />

computational viewpoint, the software engineering process wants to define a functional decomposition of the<br />

system. As an example, the SCORM object proposed by the designers to support the activity "Going into the<br />

Matter" (the HTML document "author_text1.html", see figure 5) is an "asset", which is indivisible. This type of<br />

object is only provided to the learner, without any expected feedback. But a "SCO" SCORM object is linked to it<br />

(see figure 5). This type of object contains a structure of internal objects which describes its functioning. The<br />

RM-ODP framework proposes a typology of the interfaces (ISO/IEC 10746-3, 1996, Chapter 7.1) supporting the<br />

interactions between these objects, in order to model them better:<br />

a Signal Interface, in which all the interactions are signals;<br />

an Operation Interface, in which all the interactions are operations;<br />

a Stream Interface, in which all the interactions are flows.<br />

The focus of attention of the software engineering process on the signal interface leads it to adopt the event<br />

programming paradigm as deployed by the SUN Java technology Swing (2005), for the implementation of the<br />

SCORM object.<br />

W3C<br />

<br />

XHTML 1.0<br />

<br />

<br />

XHTML 1.0 Extend Track<br />

<br />

FSL<br />

Swing<br />

<br />

HTMLDocument<br />

1<br />

<br />

JTextComponent<br />

+getUI()<br />

+getText()<br />

Figure 7: The enterprise artifact produced by the reverse engineering<br />

1<br />

FLGTextStudyElementsContentPanel<br />

JTextPane JScrollPane<br />

1 1<br />

JEditorPane<br />

1 1<br />

1<br />

1<br />

<br />

JViewPort<br />

+getExtendSize()<br />

FLGHTMLObserver<br />

1<br />

1<br />

*<br />

adlcp:scormtype<br />

imscp:file<br />

* *<br />

1<br />

<br />

JScrollBar<br />

+getVerticalScrollBar()<br />

+getViewPort()<br />

1<br />

235


Step 5: Instantiation of the technology viewpoint. From the computational viewpoint, the software engineering<br />

process needs to identify the different states and behaviors of the objects specified by designers. From a<br />

technology viewpoint, it means that the software engineering process needs to develop and integrate software<br />

captors into these objects, in order to track the behaviors of the technology objects corresponding to the<br />

information objects. In our example, it needs to identify what events (signals) could be exchanged and, for each<br />

of these events, identify the emitter and the receptor. One of the design patterns widely adopted by the software<br />

engineering community for this kind of problem is the MVC paradigm (Model/View/Control) described by<br />

Buschmann et al. (1996). It is implemented in the Java library "swing/JFC". Each object proposed by the<br />

designers and relevant to this paradigm could be a traceable object.<br />

Conclusion. The reverse engineering of a TEL system described by this use case is initiated by the design<br />

process but concerns mainly the software engineering process which, guided by the RM-ODP framework, aims<br />

to define the dynamic schema of the TEL system from its information specification. The result of the reverse<br />

engineering process is the artifact shown by figure 7. From the enterprise viewpoint, this artifact increases the<br />

quality of the specification provided by designers: even if it wasn't specified, by asking to themselves questions<br />

suggested by RM-ODP, the software engineers have enhanced the Learning Object in order to track its use. This<br />

result could be shared with designers, and generate new kinds of instructional scenarios by taking into account<br />

these tracking possibilities.<br />

The Use Case of a Reengineering of a TEL System<br />

With Chikofsky’s definition in mind (see before), one can say that reengineering a system is relevant to a model<br />

driven approach, in an iterative development process. The main reengineering acts on a TEL system are the<br />

modifications made by the designers on the informational model of the educational system, especially by taken<br />

into account the uses observed during a learning session. We will present here a reengineering use case where<br />

the analysis process provides information to the design process which is useful to understand better the real<br />

sequence of learning objects that each learner performed. With these data, both the informational and the<br />

software engineering viewpoints are modified. Figure 8 describes the four steps of the use case, one for each<br />

participating role: software engineering process role, learning process role, analysis process role, design process<br />

role.<br />

: software engineering<br />

process<br />

: produce a<br />

prototype<br />

<br />

: learning process<br />

: execute a<br />

prototype<br />

<br />

: produce an<br />

observable<br />

: design process<br />

: interpret an<br />

information<br />

Figure 8: UML Collaboration Diagram of the Reengineering Use Case<br />

Step 1: the role of the software engineering process. Each reengineering of a system implies its reverse<br />

engineering. In this use case, by applying the reverse engineering pattern (Demeyer et al., 2002), the software<br />

engineering process has developed and integrated captors to the TEL system. This allows the reverse engineering<br />

by producing information on the effects of the decisions made from the engineering viewpoint. Following the<br />

RM-ODP guidelines, the engineering objects are grouped in clusters to reduce the complexity of their<br />

manipulation. Figure 9 is an excerpt of the FreeStyle Learning deployment script on the learners’ computers.<br />

One could see that, even, from an informational viewpoint, there are several objects with a same type (e.g. 3<br />

exercises, 2 slide shows, 2 MCQ), the engineering objects are only the corresponding components (e.g. 1<br />

exercise component, 1 slide show component, 1 MCQ components).<br />

Step 2: the role of the learning process. From the computational viewpoint, these engineering objects are<br />

SCORM components (SCO) provided to each client computer. They are observed and tracked by the LMS<br />

(Learning Management System) following the SCORM technical specification. The transition/state UML<br />

diagram of figure 10 shows the global interaction between the LMS and each SCO.<br />

<br />

<br />

: analysis process<br />

: analyze an<br />

observable<br />

: produce an<br />

information<br />

236


<br />

…<br />

FreeStyle Learning<br />

…<br />

<br />

<br />

<br />

<br />

<br />

…<br />

<br />

…<br />

<br />

<br />

<br />

<br />

<br />

<br />

<br />

…<br />

<br />

<br />

SCO is launched by<br />

LMS and SCO has<br />

found the API<br />

NOTE: SCO<br />

calls Initialize()<br />

Figure 9: FreeStyle Learning Deployment Script<br />

Not<br />

Initialized<br />

GetLastError()<br />

GetErrorString()<br />

GetDiagnostic()<br />

Initialize() Terminate()<br />

Terminated<br />

"Cluster"<br />

Object<br />

"Engineering"<br />

Objects<br />

GetValue()<br />

SetValue()<br />

Commit()<br />

Running<br />

GetLastError()<br />

GetErrorString()<br />

GetDiagnostic() GetLastError()<br />

GetErrorString()<br />

GetDiagnostic()<br />

Figure 10: Transition/State UML Diagram of a SCORM component (ADL, 2004)<br />

Step 3: the role of the analysis process. The information provided by the TEL system to the analysis process is<br />

structured in data as shown in figure 11: a session ID, a date, the relevant package of the engineering object and<br />

the name of the observed event. The analysis process operates statistics with these data. It aims to discover new<br />

scenarios rather than to validate the prescribed one. This data mining structures the observed scenarios as<br />

sequences of resources, and the time spent, using the engineering viewpoint language. In our example, it leads to<br />

the identification of strong links between resources, which could be grouped in clusters and finally interpreted by<br />

designers as, for instance: “when they are expected to consult the video, many learners also accede to others<br />

resources such as questionnaires”.<br />

Step 4: the role of the design process. The design process, by its negotiation with the analysis process, aims to<br />

adapt the information model. The two processes try to find a correlation between the instances of the engineering<br />

viewpoint concepts (e.g. the new clusters defined by the analysis process) and those of the information viewpoint<br />

concepts (e.g. possible new activities supported by a new combination of resources). Figure 12 shows how the<br />

engineering and information viewpoints have been modified since the beginning of our test.<br />

Conclusion. This reengineering use case involves the main roles of the enterprise processes of a “TEL system<br />

reengineering community”. The interaction between these processes is guided by RM-ODP and leads to the<br />

237


correlation of the different viewpoints on the system. In a more general approach, the results produced by this<br />

community could be shared with other communities and increase the quality of the development of TEL<br />

systems:<br />

the engineering objects developed (e.g. the five new clusters managers) as freeware,<br />

the use case itself as an example of instantiation of IMS Learning Design,<br />

the conformity rules found during the correlation process of the different viewpoints such as knowledge and<br />

know-how which could be useful for a TEL system reengineering process.<br />

…<br />

<br />

<br />

<br />

<br />

<br />

<br />

<br />

<br />

<br />

<br />

…<br />

Conclusion<br />

Figure 11: Example of observed event provided by FreeStyle Learning<br />

Before reengineering After reengineering<br />

Information<br />

Viewpoint<br />

Introducing<br />

the LO<br />

Going into<br />

the Matter<br />

Linking Concepts<br />

and Methods<br />

Training to<br />

Understand<br />

Learning<br />

by Doing<br />

Evaluating the<br />

Acquisition<br />

Cluster<br />

Engineering<br />

Viewpoint<br />

Video<br />

Text Study<br />

Slide Show<br />

Exercise<br />

Simulator<br />

MCQ<br />

Cluster<br />

Cluster<br />

Cluster<br />

Cluster<br />

Cluster<br />

Figure 12: Information and Engineering Objects before and after the Reengineering<br />

We have presented in this paper an unique framework allowing the highlighting and the description of the<br />

transformation processes of the different models defined for the development of a TEL system. These models<br />

involve formalisms and knowledge which is specific to the relevant communities. The comprehension of these<br />

processes, their formalization and, looking forward, their instrumentation should be, for us, one of the crucial<br />

research fields in the designing of TEL systems. By supporting these model transformations through and inside<br />

different levels of abstraction and different viewpoints, the TEL community will increase its knowledge and<br />

enhance its methods for designing higher quality systems.<br />

Video<br />

Exercise<br />

Text Study Slide Show<br />

Simulator MCQ<br />

Video MCQ<br />

Text Study Slide Show<br />

Exercise<br />

Simulator<br />

Text Study<br />

MCQ<br />

Engineering<br />

Viewpoint<br />

Slide Show<br />

Information<br />

Viewpoint<br />

Discovering the<br />

Environment<br />

Introducing<br />

the LO<br />

Going into<br />

the Matter<br />

Learning<br />

by Doing<br />

Evaluating the<br />

Acquisition<br />

238


With the instantiation of the RM-ODP framework we propose on TEL, we aim to bring the design community to<br />

share and federate their experiences and expertise. Each experimentation, each deployment of a TEL system by<br />

teachers, institutions or research teams could provide benefits to the whole community. The open software<br />

community is a good example of the aims of our works, by promoting a design approach (Raymond, 2001), tools<br />

(Bar & Fogel, 2003) and communication supports (SourceForge, 2005). The designer or the user member of<br />

such a community works in a synergy with the other members by considering all the artifacts produced as<br />

prototypes, always in progress.<br />

This approach, guided by a framework as we have proposed, could be very valuable for structuring the TEL<br />

community. With this resolution in mind, we think our proposal could be the background for the emergence of a<br />

dynamic collaboration in the TEL community, in order to enhance the quality of the engineering and<br />

reengineering processes of a TEL system.<br />

References<br />

ADL. (2004). Sharable Content Object Reference Model (SCORM), 2004, (2 nd Ed), retrieved May 8, <strong>2006</strong> from<br />

http://www.adlnet.org/downloads/files/67.cfm.<br />

AndroMDA. (2005). retrieved May 8, <strong>2006</strong> from http://www.andromda.org/.<br />

Bar, M. & Fogel, B. (2003). Open Source Development with CVS, (3 rd Ed). Arizona: Paraglyph Press.<br />

Bär, H., Bauer, M., Ciupke, O., Demeyer, S., Ducasse, S., Lanza, M., Marinescu, R., Nebbe, R., Nierstrasz, O.,<br />

Przybilski, M., Richner, T., Rieger, M., Riva, C., Sassen, A.-M.., Schulz, B., Steyaert, P. & Tichelaar, S. (1999).<br />

The FAMOOS Object-Oriented Reengineering Handbook, retrieved May 8, <strong>2006</strong> from<br />

http://www.iam.unibe.ch/~famoos/handbook/4handbook.pdf.<br />

Barré, V. & Choquet, C. (2005). Language Independent Rules for Suggesting and Formalizing Observed Uses in<br />

a Pedagogical Reengineering Context. In Goodyear, P., Sampson, D. G., Yang, D. J-T., Kinshuk, Okamoto, T.,<br />

Hartley, R., Chen, N-S. (Eds.) Proceedings of the ICALT 2005 Conference, Los Alamitos, CA: IEEE Computer<br />

Society Press, 550-554.<br />

Brocke, J. (2001). Freestyle Learning - Concept, Platforms, and Applications for Individual Learning Scenarios.<br />

In Heinrich Kern (Eds.) Proceedings of 46 th International Scientific Colloquium, Ilmenau, Germany: Technical<br />

University, 149-151.<br />

Bruillard, E. & Vivet, M. (1994). Concevoir des EIAO pour des situations scolaires : approche méthodologique.<br />

Recherche en Didactique des Mathématiques, 14 (1/2), 273-302.<br />

Buschmann, F., Meunier, R., Rohnert, H., Sommerlad P. & Stal, M. (1996). Pattern-Oriented Software<br />

Architecture, <strong>Volume</strong> 1: A System of Patterns. Chichester, WS: John Wiley & Sons.<br />

Chikofsky, E. J. (1990). Reverse Engineering and Design Recovery: A Taxonomy. Software IEEE, 7, 13-17.<br />

CVS. (2005). retrieved May 8, <strong>2006</strong> from http://ximbiot.com/cvs/.<br />

Demeyer, S., Ducasse, S. & Nierstrasz, O. (2002). Object-Oriented Reengineering Patterns. San Francisco, CA:<br />

Morgan Kaufmann Publishers.<br />

Eclipse. (2005). retrieved May 8, <strong>2006</strong> from http://www.eclipse.org/.<br />

FSL. (2005). retrieved May 8, <strong>2006</strong> from http://www.wi.uni-muenster.de/aw/fsl/.<br />

GForge. (2005). retrieved May 8, <strong>2006</strong> from http://gforge.org.<br />

Grob, H. L., Bensberg, F. & Dewanto, B. L. (2004). Developing, Deploying, Using and Evaluating an Open<br />

Source Learning Management System. Journal of Computing and Information Technology, 12 (2), 127-134.<br />

239


Hummel, H., Manderveld, J., Tattersall C. & Koper, R. (2004). Educational modelling language and learning<br />

design: new opportunities for instructional reusability and personalised learning. International Journal of<br />

Learning Technology, 1 (1), 111-126.<br />

IEEE P1484.1/D9. (2001). Learning Technology Systems Architecture, Draft 9, retrieved May 8, <strong>2006</strong><br />

fromhttp://ltsc.ieee.org/wg1/files/IEEE_1484_01_D09_LTSA.pdf.<br />

IEEE 1484.12.1. (2002). Draft Standard for Learning Object Metadata, retrieved May 8, <strong>2006</strong> from<br />

http://ltsc.ieee.org/wg12/files/LOM_1484_12_1_v1_Final_Draft.pdf.<br />

Iksal, S. & Choquet, C. (2005). An Open Architecture for Usage Analysis in a E-Learning Context. In Goodyear,<br />

P., Sampson, D. G., Yang, D. J-T., Kinshuk, Okamoto, T., Hartley, R., Chen, N-S. (Eds.) Proceedings of the<br />

ICALT 2005 Conference, Los Alamitos, CA: IEEE Computer Society Press, 177-181.<br />

IMS-AF/WP. (2003). IMS Abstract Framework: White Paper, version 1.0, retrieved May 8, <strong>2006</strong> from<br />

http://www.imsglobal.org/af/afv1p0/imsafwhitepaperv1p0.html.<br />

IMS-LD/BP. (2003). IMS Learning Design Best Practice and Implementation Guide, version 1.0, Final<br />

Specification, retrieved May 8, <strong>2006</strong> from<br />

http://www.imsglobal.org/learningdesign/ldv1p0/imsld_bestv1p0.html.<br />

IMS-LD/IM. (2003). IMS Learning Design Information Model, version 1.0, Final Specification, retrieved May 8,<br />

<strong>2006</strong> from http://www.imsglobal.org/learningdesign/ldv1p0/imsld_infov1p0.html.<br />

ISO/IEC 15414. (2002). Information Technology : Open Distributed Processing Reference Model, Entreprise<br />

Language, retrieved May 8, <strong>2006</strong> from http://www.joaquin.net/ODP/DIS_15414_X.911.pdf.<br />

ISO/IEC 10746-1. (1998). Information Technology : Open Distributed Processing Reference Model, Part 1:<br />

Overview, retrieved May 8, <strong>2006</strong> from<br />

http://standards.iso.org/ittf/PubliclyAvailableStandards/c020696_ISO_IEC_10746-1_1998(E).zip.<br />

ISO/IEC 10746-2. (1996). Information Technology : Open Distributed Processing Reference Model, Part 2:<br />

Foundations, retrieved May 8, <strong>2006</strong> from<br />

http://standards.iso.org/ittf/PubliclyAvailableStandards/s018836_ISO_IEC_10746-2_1996(E).zip.<br />

ISO/IEC 10746-3. (1996). Information Technology : Open Distributed Processing Reference Model, Part 3:<br />

Architecture, retrieved May 8, <strong>2006</strong> from<br />

http://standards.iso.org/ittf/PubliclyAvailableStandards/s020697_ISO_IEC_10746-3_1996(E).zip.<br />

Koper, R. (2001). Modeling units of study from a pedagogical perspective: the pedagogical meta-model behind<br />

EML, Educational Technology Expertise Center Open University of the Netherlands, version 2, retrieved May 8,<br />

<strong>2006</strong> from http://eml.ou.nl/introduction/docs/ped-metamodel.pdf.<br />

Leinonen, T., Virtanen, O., Hakkarainen, K.& Kligyte, G. (2002), Collaborative Discovering of Key Ideas in<br />

Knowledge Building. In Stahl, G (Eds.) Proceedings of the CSCL 2002 Conference, Hillsdale, NJ: Lawrence<br />

Erlbaum Associates, 529-530.<br />

Mozilla. (2005). retrieved May 8, <strong>2006</strong> from http://www.mozilla.org.<br />

Nagase, Y., Hashimoto, D. & Sato, M. (2004). Applying Model-Driven Development to Business Systems using<br />

RM-ODP en EDOC. In Vallecillo, A., Linington, P. & Wood, B. (Eds.), Proceedings of Workshop on ODP for<br />

Enterprise Computing (WODPEC 2004), Málaga, Spain: University of Málaga, 36-42.<br />

OMG. (2001). Model Driven Architecture (MDA), retrieved May 8, <strong>2006</strong> from http://www.omg.org/mda/.<br />

OMG/ECA. (2004). Enterprise Collaboration Architecture (ECA) Specification, version 1.0, retrieved May 8,<br />

<strong>2006</strong>, from http://www.omg.org/docs/formal/04-02-01.pdf.<br />

OMG/UML. (2003). UML 2.0 Infrastructure Specification, retrieved May 8, <strong>2006</strong> from<br />

http://www.omg.org/docs/ptc/03-09-15.pdf.<br />

240


OpenLDAP05. (2005). retrieved May 8, <strong>2006</strong> from http://www.openldap.org/.<br />

OpenOffice. (2005). retrieved May 8, <strong>2006</strong> from http://www.openoffice.org/.<br />

OpenUSS. (2005). Open Source Software for Universities and Faculties (Open Source University Support<br />

System), retrieved May 8, <strong>2006</strong> from http://openuss.sourceforge.net/openuss/.<br />

Oubahssi, L., GrandBastien, M. & Claës, G. (2004). Ré-ingénierie d'une plate forme fondée sur la modélisation<br />

d'un processus global de FOAD. In Moreau, C. (Eds.), Proceedings of the TICE'04 Conference, Compiègne,<br />

France : Université de Technologie de Compiègne, 32-38.<br />

Pawlowski, J. M. (2002). Report on Quality Assurance Standards / Proposal for Future Work: Project Team<br />

Quality Assurance and Guidelines, CEN / ISSS Workshop on Learning Technologies, Working Draft, retrieved<br />

May 8, <strong>2006</strong> from http://www.wip.uni-duisburg-essen.de/imperia/md/content/elm/qadraft20020328.doc.<br />

PoseidonCE. (2005). retrieved May 8, <strong>2006</strong> from http://gentleware.com.<br />

Rawlings, A., Rosmalen, P., Koper, R., Rodríguez-Artacho, M. & Lefrere, P. (2002). Survey of Educational<br />

Modelling Languages (EMLs), CEN/ISSS WS/LT report, retrieved May 8, <strong>2006</strong> from<br />

http://www.cenorm.be/cenorm/businessdomains/businessdomains/isss/activity/emlsurveyv1.pdf.<br />

Raymond, E. S., (2001). The Cathedral and the Bazaar: Musings on Linux and Open Source by an Accidental<br />

Revolutionary, Sebastopol, CA: Edition O'Reilly.<br />

SourceForge. (2005). retrieved May 8, <strong>2006</strong> from http://www.sourceforge.net.<br />

SUN Microsystems (2003). E-Learning Framework: Technical White Paper, retrieved May 8, <strong>2006</strong> from<br />

http://www.sun.com/products-n-solutions/edu/whitepapers/pdf/framework.pdf.<br />

Swing. (2005). retrieved May 8, <strong>2006</strong> from http://java.sun.com/products/jfc/index.jsp.<br />

Tchounikine, P., Baker, M., Balacheff, N., Baron, M., Derycke, A., Guin, D., Nicaud, J.-F. & Rabardel, P.<br />

(2004). Platon-1: quelques dimensions pour l'analyse des travaux de recherche en conception d'EIAH. Paris,<br />

France: STIC-CNRS.<br />

Weka. (2005). retrieved May 8, <strong>2006</strong> from http://www.cs.waikato.ac.nz/ml/weka/.<br />

Wenger, E. (1987). Artificial Intelligence and Tutoring Systems: Computational Cognitive Approaches to the<br />

Communication of Knowledge, Los Altos, CA: Morgan Kaufmann.<br />

WUM. (2005). retrieved May 8, <strong>2006</strong> from<br />

http://hypknowsys.sourceforge.net/wiki/The_Web_Utilization_Miner_WUM.<br />

241


Karagiannidis, C. (<strong>2006</strong>). Book review: Web-Based Intelligent e-Learning Systems: Technologies and Applications<br />

(Zongmin Ma). Educational Technology & Society, 9 (4), 242-243.<br />

Web-Based Intelligent e-Learning Systems: Technologies and Applications<br />

(Book Review)<br />

Reviewer:<br />

Charalampos Karagiannidis<br />

Assistant Professor<br />

Department of Special Education<br />

University of Thessaly<br />

Volos, Greece<br />

karagian@uth.gr<br />

Textbook Details:<br />

Web-Based Intelligent e-Learning Systems: Technologies and Applications<br />

Zongmin Ma (ed.)<br />

Idea Group Inc.<br />

ISBN: 1-59140-729-X (hardcover), 1-51940-730-3 (softcover), 1-59140-731-1 (ebook)<br />

<strong>2006</strong>, 388 pages<br />

Introduction<br />

Web-Based Intelligent e-Learning Environments (WILE) attract considerable attention worldwide, since they<br />

bare the potential to improve the quality of e-learning applications and services: WILE can overcome the main<br />

shortcoming of e-learning technologies (all learners receive the same learning material, activities, etc; “one-sizefits-all”<br />

approach), by facilitating personalized learning experiences, adapted to the particular characteristics of<br />

each learner.<br />

The book focuses on the technologies and applications of WILE, aiming to present latest research, development<br />

and application results in the field.<br />

Chapter Summary<br />

The book includes 17 chapters, organized into two major sections. The first section discusses theories, key<br />

technologies and designs of WILE, while the second section covers implementation and application issues.<br />

In particular, Chapter 1 presents a framework of adaptive support for inductive reasoning ability in virtual<br />

learning environments, based on research on cognitive science; the framework accommodates adaptive<br />

navigational paths, adaptive content (amount, concreteness, structure) and adaptive information resources.<br />

Chapter 2 discusses one of the most challenging issues in WILE: adaptive authoring; it then presents a solution<br />

(MOT) based on the LAOS model, as well as a couple of WILE which have been created accordingly. Chapter 3<br />

addresses the selection and sequencing of Learning Objects (LO); it presents a utility-based model which is<br />

based on learning technologies specifications and standards, aiming to adaptively select and sequence LO,<br />

according to the learners’ profile. Chapter 4 addresses virtual learning communities (VLC); it discusses the<br />

essential criteria for their setup (social context, shared learning goal, technology and facilitation), as well as how<br />

to build culturally-inclusive VLC. Chapter 5 presents a conceptual architecture for the development of<br />

interactive educational multimedia; the architecture is open to further extensions, integration to other<br />

frameworks and standards, and adaptations to particular learner needs. Chapter 6 focuses on students’ emotions<br />

in learning environments; it reviews the state-of-the-art in this field, as well as the challenges and main<br />

difficulties in accommodating students’ affects in learning environments. Chapter 7 introduces how to use a<br />

web-based recommender system, developed with a collaborative bookmark management system approach; the<br />

system is able to effectively filter relevant resources, taking advantage of the common interests of users. Chapter<br />

8 presents MetaLinks, a domain-independent authoring tool and web server for adaptive textbooks; the tool<br />

supports active reading, which is appropriate for novice and advanced readers, and can be coherently read from a<br />

number of thematic perspectives. Chapter 9 discusses knowledge representation in ILE; they define the<br />

requirements concerning all stages of the ILE’ lifecycle, different types of users, and different system modules.<br />

Chapter 10 deals with adaptive tutoring for case-based ITS; it proposes a formal approach for the development<br />

and evaluation of ITS with reusable components. Chapter 11 addresses the use of ontologies in WILE;<br />

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the<br />

copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies<br />

are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by<br />

others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior<br />

specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.<br />

242


ontologies serve as shared knowledge representations that can be used to obtain enhanced learning object metadata<br />

records, towards the selection of LO according to learners’ profile. Chapter 12 addresses the role of learning<br />

technologies specifications and standards in WILE; it provides an overview of standards, as well as their<br />

utilization in existing systems. Chapter 13 presents AWLA - a writing e-learning alliance; AWLA is an<br />

organised set of interactive web-based utilities that, when applied in a technologically enhanced learning<br />

environment, allows learners to develop their writing skills and fulfil writing activities.<br />

In section II, Chapter 14 describes a number of interactive virtual environments (IVE) which are developed at<br />

the North Dakota State University; these IVE aim to increase student achievement and scientific problemsolving<br />

in a real-world context. Chapter 15 presents an integrated platform for educational virtual environments;<br />

the platform aims to support educational communities, synchronous online courses in multi-user 3D<br />

environments, and the creation and access of asynchronous courses through a learning content management<br />

system. Chapter 16 describes a number web-based instructional tutors that support active and engaging learning;<br />

these tutors reason about the students’ knowledge and teaching strategies, in order to adapt to them. Finally,<br />

Chapter 17 introduces the concept (and discusses the implementation and evaluation) of special purpose elearning<br />

environments (SPELE), which aim to accommodate learners’ individual learning differences.<br />

Evaluation<br />

The book includes an interesting mix of chapters, addressing both theoretical and development issues, as well as<br />

specific WILE systems and their use. Moreover, the chapters cover different aspects of WILE, including<br />

adaptive authoring, adaptive selection of learning material and activities, knowledge representation and<br />

reasoning for adaptive behaviour, etc.<br />

On the other hand, some of the chapters are not directly related to WILE - readers focused only on WILE may<br />

not be particularly interested in them. Also, the large-scale implementation and evaluation of WILE is not<br />

thoroughly covered by the book chapters - this is a very challenging issue for WILE to become mainstream.<br />

In summary, I would recommend this book to anyone who is interested in WILE. The research domain is not<br />

new (research on ITS started in 80s), however a very limited number of WILE have been tested in real, largescale<br />

implementations. The ideas presented in the book can contribute towards a better quality of e-learning<br />

systems, which can be a catalyst for the citizens of the knowledge society and knowledge-based economy.<br />

243


Glenn, L. (<strong>2006</strong>). Book review: Visual Media and the Humanities: A Pedagogy of Representation (Kecia Driver<br />

McBride). Educational Technology & Society, 9 (4), 244-245.<br />

Visual Media and the Humanities: A Pedagogy of Representation<br />

(Book Review)<br />

Reviewer:<br />

Lauren Glenn<br />

Texas A&M University – Commerce<br />

USA<br />

Tel: +1 214 771-0447<br />

lglenn@bluebottle.com<br />

Textbook Details:<br />

Visual Media and the Humanities: A Pedagogy of Representation<br />

Kecia Driver McBride, ed.<br />

2004 Knoxville: University of Tennessee Press<br />

ISBN: 1-57233-321-9<br />

18 chapters, 452 pages, introduction by McBride, bibliography, index, animations and graphics<br />

In recent scholarly discourse, the implementation of film into secondary or university courses has instigated<br />

debates regarding the usefulness of new media in relation to shifting pedagogical styles. As audiences (and by<br />

implication, students) change with the influx of visual imagery, many instructors believe that pedagogical styles<br />

need to become more pliable in relation to new media. Kecia McBride’s Visual Media and the Humanities<br />

brings together contributors from diverse university environments (Ivy League to community colleges) to<br />

discuss the importance of teaching students to be critical and self-reflective when reading texts and images in<br />

humanities courses that use new media, including film, internet, videoconferencing, and digitalized texts.<br />

McBride’s collection includes several articles that explore the impact of new modes of visual representation on<br />

pedagogical styles while emphasizing the importance of teaching students to incite reflective responses as they<br />

read images and texts. In “‘Where can I get a camera?’: Documentary Film, Visual Literacy, and the Teaching<br />

of Writing,” Dale Jacobs claims that visual literacy has been underestimated in the school environment, pointing<br />

to his earliest encounter with visual imagery and literacy through Sesame Street episodes which taught him the<br />

basic structure of visual literacy. He claims, “The unfortunate situation is that what I learned in front of that<br />

television set on those countless mornings – the rudiments of visual literacy, or how to read and make meaning<br />

from moving visual images – is often discounted or devalued in the school environment” (156). Although<br />

contemporary Americans are bombarded with visual imagery from childhood on, Diane Negra and Walter Metz<br />

point out, “Americans are used to articulating codes of pleasure but not codes of meaning in their texts” (229).<br />

Negra and Metz present an example of using media in introductory college courses to combat the American<br />

public’s lack of critical analysis when confronted with visual images in television and film. James W. Miles II<br />

claims that the higher a student advances in academia (into secondary classes or college courses), the more the<br />

“silent codex,” or written final product, is emphasized through book reports, essays, and the written discourse<br />

(75). He argues that such practices may not hold their value as rhetorical demands evolve in society. The<br />

articles included in this book do not argue for the abandonment of all traditional pedagogical methods, but rather<br />

for a malleability of teaching conventions as time changes the emphasis on visual images in society.<br />

Research in the twenty first century is changing rapidly, and nowhere is it more apparent than through the use of<br />

the Internet. Thomas Crochunis’ article, “Visualizing Feminist Theater History on the Web,” discusses how<br />

web-based working groups can offer scholarly support in areas of limited and on-going scholarship for students<br />

and scholars. In “Simulation Machines, Media Boundaries, and the Re-expansion of Composition,” James W.<br />

Miles II points to recent advancements in voice recognition technology and Internet communication to predict<br />

that student writing will become radically more conversational in the future. He suggests that educators use this<br />

change in technology and style to alter pedagogy in composition courses by comparing written and spoken<br />

essays in class.<br />

Contributors introduce topics such as culture, history, argument, foreign language, technology, and web-based<br />

groups. When cultural or historical issues become distanced from students in the classroom, visual images may<br />

be capable of influencing learning or instigating discussions in new ways. In “The World in a Frame:<br />

Introducing Culture through Film,” Gerald Duchovnay approaches teaching a capstone university course that<br />

initiates students into cultural issues. He provides a resourceful example demonstrating how students can be<br />

introduced to cultural differences through film, guest lecturers, and supplementary materials. The article outlines<br />

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the<br />

copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies<br />

are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by<br />

others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior<br />

specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.<br />

244


a sample course focus, textbooks, films, and assignments that would be beneficial to anyone wanting to broach<br />

cultural issues in the classroom.<br />

While recognizing the changing emphasis on visual literacy in contemporary society, many of the articles<br />

provide analysis of specific films and their usefulness in classroom pedagogy. Maria Bachman, for example,<br />

demonstrates how she uses Pleasantville in conjunction with Plato’s “Allegory of the Cave” to teach students<br />

that higher knowledge consists of educators guiding students to unearth their own questioning abilities.<br />

Likewise, in “An Imitation of Life: Deconstructing Racial Stereotypes in Popular Culture,” Stephanie Thompson<br />

outlines her approach to initiating classroom discussion on the “perpetuation of racist ideology” by utilizing<br />

Fannie Hurst’s novel Imitation of Life with two film adaptations (1934 and 1959) to demonstrate the evolving<br />

racial issues in Western society (205) in her introduction to composition and women’s studies course.<br />

From classic films to contemporary films, Internet groups to web-based writing styles, McBride’s collection<br />

embraces various approaches to using new media to teach in an academic environment that relies heavily on<br />

visual literacy. Anyone interested in accessible scholarly approaches aimed at utilizing new media as a learning<br />

tool in diverse classroom settings will find this collection of course outlines, discussion of implementation, and<br />

pedagogical examples of contemporary classroom issues to be an invaluable resource.<br />

245

Hooray! Your file is uploaded and ready to be published.

Saved successfully!

Ooh no, something went wrong!