Complete issue in PDF - Educational Technology & Society

Journal of 

Educational Technology & 

Society 

Published by 

International Forum of Educational Technology & Society 

Endorsed by 

IEEE Learning Technology Task Force 

January 2004 

Volume 7 Number 1 

ISSN: 1176-3647

Educational Technology & Society 

An International Journal 

Aims and Scope 

Educational Technology & Society is a quarterly journal published in January, April, July and October. Educational Technology & 

Society seeks academic articles on the issues affecting the developers of educational systems and educators who implement and manage such 

systems. The articles should discuss the perspectives of both communities and their relation to each other: 

• Educators aim to use technology to enhance individual learning as well as to achieve widespread education and expect the technology to 

blend with their individual approach to instruction. However, most educators are not fully aware of the benefits that may be obtained by 

proactively harnessing the available technologies and how they might be able to influence further developments through systematic 

feedback and suggestions. 

• Educational system developers and artificial intelligence (AI) researchers are sometimes unaware of the needs and requirements of typical 

teachers, with a possible exception of those in the computer science domain. In transferring the notion of a 'user' from the humancomputer 

interaction studies and assigning it to the 'student', the educator's role as the 'implementer/ manager/ user' of the technology has 

been forgotten. 

The aim of the journal is to help them better understand each other's role in the overall process of education and how they may support 

each other. The articles should be original, unpublished, and not in consideration for publication elsewhere at the time of submission to 

Educational Technology & Society and three months thereafter. 

The scope of the journal is broad. Following list of topics is considered to be within the scope of the journal: 

Architectures for Educational Technology Systems, Computer-Mediated Communication, Cooperative/ Collaborative Learning and 

Environments, Cultural Issues in Educational System development, Didactic/ Pedagogical Issues and Teaching/Learning Strategies, Distance 

Education/Learning, Distance Learning Systems, Distributed Learning Environments, Educational Multimedia, Evaluation, Human- 

Computer Interface (HCI) Issues, Hypermedia Systems/ Applications, Intelligent Learning/ Tutoring Environments, Interactive Learning 

Environments, Learning by Doing, Methodologies for Development of Educational Technology Systems, Multimedia Systems/ Applications, 

Network-Based Learning Environments, Online Education, Simulations for Learning, Web Based Instruction/ Training 

Editors 

Kinshuk, Massey University, New Zealand; Ashok Patel, CAL Research & Software Engineering Centre, UK; Reinhard Oppermann, 

Fraunhofer Institut Angewandte Informationstechnik , Germany. 

Associate editors 

Alexandra I. Cristea, Technical University Eindhoven, The Netherlands; John Eklund, Access Australia Co-operative Multimedia 

Centre, Australia; Vladimir A Fomichov, K. E. Tsiolkovsky Russian State Tech Univ, Russia; Olga S Fomichova, Studio "Culture, 

Ecology, and Foreign Languages", Russia; Piet Kommers, University of Twente, The Netherlands; Chul-Hwan Lee, Inchon National 

University of Education, Korea; Brent Muirhead, University of Phoenix Online, USA; Scott Overmyer, Massey University, New 

Zealand; Demetrios G Sampson, University of Piraeus and ITI-CERTH, Greece; Erkki Sutinen, University of Joensuu, Finland; 

Vladimir Uskov, Bradley University, USA. 

Advisory board 

Ignacio Aedo, Universidad Carlos III de Madrid, Spain; Rosa Maria Bottino, Consiglio Nazionale delle Ricerche, Italy; Tak-Wai 

Chan, National Central University, Taiwan; Nian-Shing Chen, National Sun Yat-senUniversity, Taiwan; Grainne Conole, 

Southampton University, UK; Roger Hartley, Leeds University, UK; J R Isaac, National Institute of Information Technology, India; 

Akihiro Kashihara, The University of Electro-Communications, Japan; Ruddy Lelouche, Universite Laval, Canada; David Merrill, 

Utah State University, USA; Marcelo Milrad, Växjö University, Sweden; Riichiro Mizoguchi, Osaka University, Japan; Toshio 

Okamoto, The University of Electro-Communications, Japan; Brian K. Smith, Pennsylvania State University, USA; J. Michael 

Spector, Syracuse University, USA. 

Assistant Editors 

Sheng-Wen Hsieh, National Sun Yat-sen University, Taiwan; Taiyu Lin, Massey University, New Zealand; Kathleen Luchini, 

University of Michigan, USA; Nikos Manouselis, University of Piraeus and ITI-CERTH, Greece; Dorota Mularczyk, Independent 

Researcher & Web Designer; Carmen Padrón Nápoles, Universidad Carlos III de Madrid, Spain; Ali Fawaz Shareef, Massey 

University, New Zealand; Jarkko Suhonen, University of Joensuu, Finland. 

Executive peer-reviewers (English section) 

http://ifets.ieee.org/periodical/peer_reviewers.html 

Subscription Prices and Ordering Information 

Institutions: NZ$ 150 (~ US$ 75) per year (four issues) including postage and handling. 

Individuals (no school or libraries): NZ$ 100 (~ US$ 50) per year (four issues) including postage and handling. 

Single issues (individuals only): NZ$ 35 (~ US$ 18) including postage and handling. 

Subscription orders should be sent to The International Forum of Educational Technology & Society (IFETS), c/o Prof. Kinshuk, 

Information Systems Department, Massey University, Private Bag 11-222, Palmerston North, New Zealand. Tel: +64 6 350 5799 ext 2090. 

Fax: +64 6 350 5725. E-mail: kinshuk@ieee.org. 

Advertisements 

Educational Technology & Society accepts advertisement of products and services of direct interest and usefulness to the readers of the 

journal, those involved in education and educational technology. Contact the editors at kinshuk@ieee.org. 

Abstracting and Indexing 

Educational Technology & Society is abstracted and indexed in ACM Guide to Computing Literature, Computing Reviews, DBLP, 

Educational Administration Abstracts, Educational Research Abstracts, ERIC Clearinghouse on Information & Technology, Inspec, 

Technical Education & Training Abstracts, and VOCED. 

Sponsorship 

Educational Technology & Society acknowledges the generous financial sponsorship provided by Centre for Research and Technology - 

Hellas, Informatics and Telematics Institute (CERTH/ITI), Greece towards the establishment of the print version of the journal. 

ISSN 1436-4522 1436-4522. (online) © International and 1176-3647 Forum (print). of Educational © International Technology Forum of & Educational Society (IFETS). Technology The authors & Society and (IFETS). the forum The jointly authors retain and the copyright forum jointly of the retain articles. the 

copyright Permission of the to make articles. digital Permission or hard copies to make of digital part or or all hard of this copies work of for part personal or all of or this classroom work for use personal is granted or without classroom fee use provided is granted that without copies are fee not provided made or that distributed copies 

are for not profit made or or commercial distributed advantage for profit and or commercial that copies advantage bear the full and citation that copies on the bear first the page. full Copyrights citation on the for components first page. Copyrights of this work for owned components by others of this than work IFETS owned must by be 

others honoured. than IFETS Abstracting must with be honoured. credit is permitted. Abstracting To with copy credit otherwise, is permitted. to republish, To copy to otherwise, post on servers, to republish, or to redistribute to post on to servers, lists, requires or to redistribute prior specific to lists, permission requires and/or prior a 

specific fee. Request permission permissions and/or a from fee. the Request editors permissions at kinshuk@massey.ac.nz. 

from the editors at kinshuk@ieee.org. 

i

Guidelines for authors 

Submissions are invited in the following categories: 

• Peer reviewed publications: a) Full length articles (4000 - 7000 words), b) Short articles, Critiques and Case studies (up to 3000 words) 

• Book reviews 

• Software reviews 

• Website reviews 

All peer review publications will be refereed in double-blind review process by at least two international reviewers with expertise in the 

relevant subject area. Book, Software and Website Reviews will not be reviewed, but the editors reserve the right to refuse or edit review. 

• Each peer review submission should have at least following items: title (up to 10 words), complete communication details of ALL 

authors , an informative abstract (75-200 words) presenting the main points of the paper and the author's conclusions, four - five 

descriptive keywords, main body of paper (in 10 point font), conclusion, references. 

• Submissions should be single spaced. 

• Footnotes and endnotes are not accepted, all such information should be included in main text. 

• The paragraphs should not be indented. There should be one line space between consecutive paragraphs. 

• There should be single space between full stop of previous sentence and first word of next sentence in a paragraph. 

• The keywords (just after the abstract) should be separated by comma, and each keyword phrase should have initial caps (for example, 

Internet based system, Distance learning). 

• Do not use 'underline' to highlight text. Use 'italic' instead. 

Headings 

Articles should be subdivided into unnumbered sections, using short, meaningful sub-headings. Please use only two level headings as far 

as possible. Use 'Heading 1' and 'Heading 2' styles of your word processor's template to indicate them. If that is not possible, use 12 point 

bold for first level headings and 10 point bold for second level heading. If you must use third level headings, use 10 point italic for this 

purpose. There should be one blank line after each heading and two blank lines before each heading (except when two headings are 

consecutive, there should be one blank like between them). 

Tables 

Tables should be included in the text at appropriate places and centered horizontally. Captions (maximum 6 to 8 words each) must be 

provided for every table (below the table) and must be referenced in the text. 

Figures 

Figures should be included in the text at appropriate places and centered horizontally. Captions (maximum 6 to 8 words each) must be 

provided for every figure (below the figure) and must be referenced in the text. The figures must NOT be larger than 500 pixels in width. 

Please also provide all figures separately (besides embedding them in the text). 

References 

• All references should be listed in alphabetical order at the end of the article under the heading 'References'. 

• All references must be cited in the article using "authors (year)" style e.g. Merrill & Twitchell (1994) or "(authors1, year1; authors2, 

year2)" style e.g. (Merrill, 1999; Kommers et al., 1997). 

• Do not use numbering style to cite the reference in the text e.g. "this was done in this way and was found successful [23]." 

• It is important to provide complete information in references. Please follow the patterns below: 

Journal article 

Laszlo, A. & Castro, K. (1995). Technology and values: Interactive learning environments for future generations. Educational Technology, 

35 (2), 7-13. 

Newspaper article 

Blunkett, D. (1998). Cash for Competence. Times Educational Supplement, July 24, 1998, 15. 

Or 

Clark, E. (1999). There'll never be enough bandwidth. Personal Computer World, July 26, 1999, 

http://www.vnunet.co.uk/News/88174. 

Book (authored or edited) 

Brown, S. & McIntyre, D. (1993). Making sense of Teaching, Buckingham: Open University. 

Chapter in book/proceedings 

Malone, T. W. (1984). Toward a theory of intrinsically motivating instruction. In Walker, D. F. & Hess, R. D. (Eds.) Instructional 

software: principles and perspectives for design and use, California: Wadsworth Publishing Company, 68-95. 

Internet reference 

Fulton, J. C. (1996). Writing assignment as windows, not walls: enlivening unboundedness through boundaries, retrieved July 7, 2002 

from http://leahi.kcc.hawaii.edu/org/tcc-conf96/fulton.html. 

Submission procedure 

Authors, submitting articles for a particular special issue, should send their submissions directly to the appropriate Guest Editor. Guest 

Editors will advise the authors regarding submission procedure for the final version. 

All submissions should be in electronic form. The editors will acknowledge the receipt of submission as soon as possible. 

The preferred formats for submission are Word document and RTF, but editors will try their best for other formats too. For figures, GIF 

and JPEG (JPG) are the preferred formats. Authors must supply separate figures in one of these formats besides embedding in text. 

Please provide following details with each submission: Author(s) full name(s) including title(s), Name of corresponding author, Job 

title(s), Organisation(s), Full contact details of ALL authors including email address, postal address, telephone and fax numbers. 

The submissions should be sent via email to (Subject: Submission for Educational Technology & Society journal): kinshuk@ieee.org. 







ii

Journal of Educational Technology & Society 

Volume 7 Number 1 2004 

Formal discussion summaries 

Table of contents 

The E-book vs. the ordinary book 

Moderator and Summarizer: Zygmunt Scheidlinger 

Should Online Course Design Meet Accessibility Standards? 

Moderator and Summarizer: Peter Paolucci 

Interactivity in computer-mediated college and university education: A 

recent review of the literature 

Moderator and Summarizer: Brent Muirhead and Charles Juwah 

Full length articles 

Teacher Training as Collaborative Problem Solving 

Jorge Barojas 

An Online Assistant for Remote, Distributed Critiquing of Electronically Submitted 

Assessment 

Penny Baillie-de Byl 

Modeling Web-based Educational Systems: Process Design Teaching Model 

Franca Pantano Rokou, Elena Rokou and Yannis Rokos 

General Practitioners and Online Continuing Professional Education: Projected 

Understandings 

Jan Brace-Govan and Mark Gabbott 

The Effects of a Constructivist Intervention on Pre-Service Teachers 

Kathryn DiPietro 

Emerging Online Learning Environments and Student Learning: An Analysis of Faculty 

Perceptions 

Carrie B. Myers, Dennis Bennett, Gary Brown and Tom Henderson 

Preparing for Distance Learning: Designing An Online Student Orientation Course 

Jane Bozarth, Diane D. Chapman and Laura LaMonica 

ISSN 1436-4522 1436-4522. (online) © International and 1176-3647 Forum (print). of Educational © International Technology Forum & of Society Educational (IFETS). Technology The authors & Society and the (IFETS). forum The jointly authors retain and the the copyright forum jointly of the retain articles. the 

Permission copyright of to the make articles. digital Permission or hard copies to make of part digital or all or of hard this copies work for of part personal or all or of classroom this work use for is personal granted or without classroom fee provided use is granted that copies without are fee not provided made or that distributed copies 

are for profit not made or commercial or distributed advantage for profit and or that commercial copies bear advantage the full and citation that copies on the bear first page. the full Copyrights citation on for the components first page. Copyrights of this work for owned components by others of than this work IFETS owned must by be 

others honoured. than Abstracting IFETS must with be honoured. credit is permitted. Abstracting To with copy credit otherwise, is permitted. to republish, To copy to post otherwise, on servers, to republish, or to redistribute to post on to lists, servers, requires or to prior redistribute specific to permission lists, requires and/or prior a 

specific fee. Request permission permissions and/or from a fee. the Request editors permissions at kinshuk@massey.ac.nz. 


1-5 

6-11 

12-20 

21-28 

29-41 

42-50 

51-62 

63-77 

78-86 

87-106 

iii

Higher education staff experiences of using web-based learning technologies 

Debra Salmon and Mathew Jones 

Cohort Learning Online in Graduate Higher Education: Constructing Knowledge in Cyber 

Community 

Elizabeth J. Tisdell, Gabriele I. E. Strohschen, Mary Lynn Carver, Pam Corrigan, 

Janet Nash, Mary Nelson, Mike Royer, Robin Strom-Mackey and Marguerite O'Connor 

Book reviews 

Instructional Design in the Real World: A View from the Trenches 

Reviewer: Stephen Corich 

Software reviews 

Review of the Tools for the Cognitive Task Analysis 

Reviewer: Youngmin Lee 







107-114 

115-127 

128-129 

130-139 

iv

Scheidlinger, Z. (2004). The E-book vs. the ordinary book. Educational Technology & Society, 7 (1), 1-5. 

Moderator & Sumamrizer: 

Zygmunt Scheidlinger 

Tel Aviv, Israel 

zscheidl@zahav.net.il 

Discussion Schedule: 

Discussion: 11-20 August 2003 

Summing-up: 21-22 August 2003 

Pre-Discussion Paper 

The E-book vs. the ordinary book 

The fast progress in information technology brings us to the point where we should analyze the position of a 

commercially printed paper book in the near future. One may state unequivocally that the printed book created 

by Gutenberg (1398-1468) has finished its role as a carrier of ideas and now it has no future, just as it was with 

the wonderfully illuminated fables written by monks in the monasteries. The revolution brought by the computer 

surpasses by far that introduced by the invention of printing. 

As is known to everybody a book consists of subject matter and its physical basis – printing paper, imposed type 

and cover. The computer may affect, to some extent, the subject matter of a book, but will never replace the 

essence of the book and nobody is considering “replacing the book by a computer” - but it is clear that replacing 

the existing form of the book by an electronic book is inevitable and will occur very soon. Some of the obvious 

advantages of the e-book are recapitulated in the following: 

1. The ordinary printed book is produced in a large number of copies. This calls for a large initial investment, 

for providing storage and accounting for the remaining stock, while an E-book is produced in the exact 

quantities requested by the market. 

2. The distribution of an e-book is practically free of charge, and its delivery is immediate - by electronic mail. 

3. The e-book may be updated as often as it is necessary while an ordinary printed book becomes obsolete, 

sometimes very fast. 

4. The e-book may utilize any type of font in whatever size, and may use an electronic magnifying glass that 

shows the relevant part of text with any requested magnification. 

5. The e-book may be downloaded on a floppy disk, CD or any other storage device that occupies much less 

space and weights much less than a printed book. All encyclopedias and many other reference materials are 

already printed in form of e-book. 

6. The author’s honorary may be paid after selling every copy. 

7. The buyer of an e-book may print it at home in any format. He may print only the interesting chapters or 

parts of the book, and may share the book with an unlimited number of friends. 

8. The author of an e-book sees it published immediately after the book has been finished and corrected. 

9. The e-book may be translated into several languages immediately and free of charge. 

10. All these features may contribute to a substantial increase of the number of the readers. 

11. One may reasonably expect that in course of the next 3 to 5 years storage devices with memory of some 50 

to 75 GB will appear, enabling users to store a whole library containing several thousand books on a single 

disk. Thus every child may be supplied with all the masterpieces of world literature on a single disk. 

12. An e-book may contain large number of illustrations, photos and diagrams without increasing its weight or 

volume. 

13. An e-book may be read by voice, as well as contain clips of music or poetry. 

14. There is no delay between writing an e-book and its publishing worldwide. 

15. The “transportation” of an e-book does not call for packing and handling. 

16. An e-book may be printed on some kind of durable medium which may survive hundreds of years, even 

under adverse climatic conditions. 

17. Conducting a search for specific passages or for certain information in an e-book is decisively easier than in a 

commercially printed book. 

18. The fate of an old book is, in most cases, regretfully - that it gathers dust on the shelf until somebody will 

dare to throw it away. Most people don't have enough time to read new stuff, let alone the old one. In the case 

of an e-book the problem does not exist, since it occupies a miniscule space on the storage medium and may 

be deleted by one click without ecological damage. 

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the 

copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies 

are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by 

others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior 

specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 

1

19. An e-book provides access to most of the museums and thus may contribute to the dissemination of art and 

sciences among the population. The existing catalogues of most museums are heavy and costly while the 

quality of pictures is, in general, far below what a good modern monitor may provide. 

20. Undoubtedly using computer graphics, sound and animation teaching kids read and write may be performed 

more effectively than by using old printed books only (in Poland the book teaching read and write – 

Elementarz Falskiego- was in use for almost hundred years). 

21. Computer-based learning creates lasting interest in learning, while oral teaching associated with 

commercially printed books is deadly boring. Many teachers feel that things learned from computer stay 

etched in the mind for a long time if not forever, while knowledge gathered from old handbooks evaporates 

very fast. 

22. Computers allow an extra dimension in education (of adults and children) and call for a complete change of 

philosophy and teaching methods. Such a change may take place only by the replacement of the entire 

structure of education by another one, fully computer-based. 

23. The Introduction of computers in primary schools helps to introduce this form of media earlier in life, and 

there is no doubt that this is the most valuable aid in further education. 

24. There is no place for a distinction between e-books and commercially printed books since we can place 

books online and download their contents in minutes. Books and computers are made for each other and 

serve each other’s purposes. But where computers are really going to shine is in conjunction with the Internet 

in online courses of education. 

25. The e-book provides easy links to similar articles while commercially printed material besides indicating (in 

the best case) the existence of relevant material - does not assist in getting it. 

26. The e-book can offer interactivity that engages the student. The e-book and the computer have the potential 

of analyzing the students learning pattern and optimize the learning process. Frequently the computer 

(especially the Internet with its hypertext links) allows the user to find information without effort. All this 

proves that the e-book has the potential of surpassing (rather than encompassing) commercially printed 

books. 

27. Artificially intelligent, high quality courseware allows tailoring lessons individually for each child – nothing 

similar can be provided by a commercially printed handbook. 

28. One picture is worth thousand words – the number of illustrations in an e-book is practically unlimited and 

the cost of their insertion is miniscule. 

29. An e-book provides an opportunity for a personal contact with its author. 

30. An e-book provides all the means for supervising the child's activities and reactions, as well as his interests 

and this way the learner can benefit from the absorbed material. 

31. Excellent multimedia material stimulates the child’s intellect more than anything else. It induces further 

reading or watching, and may serve as the most powerful source of knowledge about the world and about 

people. The wealth of information provided by the e-book cannot be compared with that supplied by ordinary 

paper book. 

32. The e-book stimulates the development of artistic talents of children, especially in the field of graphic art, 

drama, choreography, architecture, literature and music. 

33. The reduction of the price of printers (a very good printer produced by HP costs below $40), together with 

reduction of the price of ink cartridges enables inexpensive printing of large blocks of downloaded materials, 

including e-books. 

34. Reading material can be bound together and the result will not differ from a book produced by a professional 

book publisher at a much lower price. 

35. In case of an e-book the attention of the child is concentrated on the text and the accompanying graphics 

while the dreadfully boring character of the paper book causes that the child is constantly distracted. A well 

developed, attractive courseware, prepared by team of outstanding pedagogues will undoubtedly raise 

standards among all learners, including primary school children. 

36. Basing on the experience of the last 45 years there is no doubt that the next decade will bring processors that 

are, at least, ten times faster and hard drives that are at least ten times bigger, with all the possibilities arising 

from such a development in the education process. Enormous R&D budgets and thousands of talented 

scientists/engineers dealing with the subject guarantee that such a development is highly probable. 

37. Valuable, well researched information is presented on the screen of TV or computer, because it attracts 

viewers and raises rating, while a paper book rarely contains information that is given away for nothing. 

Besides, in most cases this information is worthless, obsolete or outdated. 

38. Reading the paper book goes hand in hand with headaches, eye strain and mobility problems because of the 

font of small size, unsuitable for people above 50, poor paper and the bleak printing ink (you see the font 

from the other side of the page), while the computer enables to get a font of any size by using the inherent 

features of the computer and, in addition, has the wonderful ”electronic magnifier glass” feature that enables 

reading of every text even by people with very weak sight, without strain. 

2

39. By using computers in an interactive way the horizons are virtually limitless. Of course we need to 

completely rethink teaching methods and to redefine the role of the teacher, if any. 

40. If we accept the idea that it is the subject matter that counts, not the decorative effect, then the e-book has all 

the advantages – a decisively lower price, immediate delivery, easy transportation in case of relocation, no 

load on the floor, no fire hazard, fast access to needed information, a possibility of sharing with friends, easy 

disposal of outdated books and many others. 

41. Of course such issues as the cost of a computer, portability and reliability will have to be resolved – these 

problems do not seem insurmountable – before e-books will finally replace ordinary books and mediocre 

teachers as the main tool for education. 

42. An e-book may be discussed with a group of friends scattered over the globe while discussion of an ordinary 

book calls for the gathering of participants in one place or arranging a costly teleconference. 

43. The failure of introducing of e-books and computers into the educational process so far results from the 

limited courseware development budget, from the lack of knowledge of most educators how to fully utilise 

computers, and last but not least, from the fear that computer will partially or fully replace mediocre teachers. 

44. An ordinary book may be destroyed by fire or water and nobody keeps a back-up for books. However, an ebook 

may be stored on a CD-ROM less than 1mm thick and one may easily have a back-up that will cost a 

few cents and be kept in a safe place. 

45. The delivery of ordinary book lasts several days and the cost of delivery is very substantial (comparable with 

the cost of the book itself) while an e-book is delivered in minutes and it costs practically nothing. 

46. Ordinary books are often large and heavy - e-books in hundreds may be loaded on a single CD. 

47. An e-book enables to get explanations, interpretations, and bibliographical data. It may contain a Thesaurus 

of words and phrases and thus assure better understanding of the subject matter. 

48. Only the e-book together with distance learning may satisfy the growing number of learners that for the age 

span 14 to 17 exceeds 950 per thousand and the people in their thirties and forties that encounter new 

methods, new machines, new materials and new techniques. 

49. Only a transition to e-book enables the utilization of the constructivist learning theory that is the leading 

learning theory today. 

50. The e-book may prepare the learner to learn independently, without a teacher – in a real situation. The e-book 

may provide the learner with all the knowledge that is requested for understanding the new material. Only an 

e-book may contain the answers to all possible questions of the learner and may provide all the information 

that may be necessary for understanding the new material so that the presence of the teacher/lecturer should 

be redundant. 

Bibliography 

Scheidlinger, Z. (1999). Education calls for a new philosophy. Educational Technology & Society, 2 (3), 119- 

122. 

Post-discussion Summary 

1. Introduction 

The discussion was centered on the long standing emotions and deeply based prejudices regarding the existing 

form of the paper book as something eternal, given from heaven and pointing to the cultural level of the owner. 

Such an attitude to the book and to the private library persisted during the eighteen and nineteen century. 

In recent five to ten years this attitude started to change. Encyclopedias and reference books appear almost 

entirely in form of laser disks, the capacity of which steadily increases, reaching at present tens of gigabytes and, 

most probably, toward the end of a decade, will reach hundred or even thousand of gigabytes. The efforts to 

increase the capacity of disks are partially dictated by the DVD popularity, which captures the growing share of 

the market and calls for disks of large capacity. 

The posting by Bowerbird Intelligentleman covers most of the problems connected with the transition from 

paper book to the e-book. It calls for creation of high quality e-book software with the following features: 

1. Ability to have several pages/books open simultaneously; 

2. Ability to highlight portions of the text and variety of annotations capabilities such as margin notes, 

underlining, dog ears, bookmarks, “paper clipping”. 

3

3. Ability to provide font of any kind, any convenient size, any color, any leading between the lines, any 

convenient background color. 

4. Because of the abovementioned abilities the eye strain may be reduced or even totally avoided. 

5. Because of the possibility of using wireless keyboard and mouse, together with voice recognition, the 

posture when reading and writing may be changed within wide limits, much wider than in case of paper 

book. 

6. Ability to store an equivalent of thousands of volumes (or even tens of thousands of volumes in the near 

future). 

7. Dimensions and weight of the most recent readers (Scheidlinger, 1999) do not exceed those of a paperback. 

8. Ability to work in landscape or portrait format – whatever suits the reader. 

9. Ability to resize the window with rewrapping the text accordingly 

10. Ability to toggle the horizontal justification of lines. 

11. Ability to toggle vertical justification (page balancing). 

12. Ability to have the user read out loud to him. 

13. Ability to control the widow/orphan of the text in the window. 

14. Ability to choose all lower-case display. 

15. Ability to jump to any page directly, by page number. 

16. Ability to return easily to the last page displayed. 

17. Ability to jump, upon opening the book, to the page last shown. 

18. Ability to show all the time the relative position in the book. 

19. Ability to search the text for a key term (i.e. word or phrase) 

20. Ability to conduct multiple key term searches simultaneously. 

21. Ability to show the lines containing any found key term. 

22. Ability to write results of key term searches to the clipboard. 

23. 23 Ability to write results of key term searches to the file. 

24. Ability to copy any picture in the e-book. 

25. Ability to write out any sound in the e-book. 

26. Ability to copy any movie in the e-book to clipboard. 

27. Ability to make text annotations at any point in the book. 

28. Ability to make various types of marks anywhere in the book. 

29. Ability to create easily e-books using their own text. 

Jean-Marc Dubois sees many serious hurdles before paper books may be replaced by e-books. The main obstacle 

is the necessity of a radical change in the whole education philosophy. 

He does not take into account the great number of highest quality specialists mobilized for the task, enormous 

budget at the disposal of the task, the most valuable experience gathered in course of development of complex, 

multi - transistor chips. 

Cooperation between specialists located at different places on the globe may enhance progress of the necessary 

technology. 

Progress in the cognitive theory achieved in recent decade provides a basis for new effective teaching methods 

and therefore comparison with the failures in the past of switching over to video tapes and later to CDs should be 

considered in this case as irrelevant. 

As soon as computer will be given an active role, and replace, at least partially, the teacher, successful 

computerization of the education process will proceed very fast. 

Charles Adams points out that small e-book readers may call for frequent page changes thus slowing down the 

reading. She also thinks that ordinary books make addition of notes and finding them again simpler than in ebook. 

Mary Harsh reports about natural language agents developed by her that could be accessed through a hotlink in 

an online web environment. She intends to use it in combination with web-based educational materials in gamelike 

activities. Students may ask any question not just a list of pre-defined questions. The agent is built on 

Filemaker Pro with CDML web components. It has the capability to be edited from the web as well as queried 

from the web. Natural language agents are not as totally spontaneous as real human beings but contrary to 

humans are available 24/7/365 and provide instant feedback. 

4

Errol Thompson asserts that searching, indexing and hyperlinking attract him to use e-learning tools but he sees 

a possibility of employing them for printed books too. 

If we look for defender of the idea that the future belongs to e-book Manny Halpern provides the necessary 

arguments. She finds it easier to open several windows than to open several paper books. In case of relocation – 

quite often in many countries –it is easier to move around e-books than tons of paper books and therefore she 

stopped purchasing text books. If the purpose is to compile information and create new information the ability to 

cut, paste and edit from several electronic sources is decisively easier than from paper books. 

For preparation a chapter illustrating dynamic work activities she found that no still picture, let alone words, 

could have adequately explained what actually takes place. A video-clip inserted into the text was found much 

better for this purpose. She did not found that reading text on the screen causes more eye strain than paper 

format. She considers that the ability to control display offered by e-book was her salvation. What may cause eye 

strain for one, may be a blessing for the visually impaired. She informs that flexible, deformable displays are 

already available (Fildes, 2003). 

Scheidlinger, Z. (1999). Education calls for a new philosophy. Educational Technology & Society, 2 (3), 119- 

122. 

Fildes, J. (2003). Hi-Tech tome takes on paperbacks. BBC News, Retrieved January 17, 2004 from: 

http://news.bbc.co.uk/2/hi/technology/3173835.stm. 

5

Paolucci, P. (2004). Should Online Course Design Meet Accessibility Standards? Educational Technology & Society, 7 (1), 

6-11. 

Should Online Course Design Meet Accessibility Standards? 


Peter Paolucci 

York University and Learn Canada, Canada 

paolucci@learncanada.org 


Discussion: October 6-15, 2003 

Summing-up: October 16-17, 2003 


Slip-Slidin' Away: The Importance of Standards and the Absence of a Definitive Standard 

For quite some time now, the Internet and its content have been developing in a haphazard way. This is not 

inherently a bad thing and perhaps it's even unavoidable, but it does have its problems. Web-based languages 

such as HTML, JavaScript, XHTML, XML, Perl, etc have been standardized into versions, but these standards 

shift over time because these languages are almost all very new and in many cases, current online content does 

not conform to them anyway as long as browsers continue to interpret bad or proprietary code in the way that 

designers want. Thus it has come to pass that a considerable amount of current online content "works" because 

browsers display what we want them to display, but the code underneath that creates that display is irregular, 

inconsistent, often idiosyncratic, and in many cases, produced according to the biases of the vendor. Moreover, 

interface design itself has tended to operate on conventions (unconsciously accepted practices) rather than 

standards (methodically crafted and tested practices). Frames for example, were used extensively up until a few 

years ago when they fell into disrepute. The convenience of having a standard omni-present navigational panel 

was eventually outweighed by the fact that neophytes were unable to bookmark anything except the frameset file 

(the splash page of the website). 

The browsers themselves, which were originally designed to interpret HTML code (and now other web 

languages such as JavaScript, XML and XHTML) are by no means uniform or consistent. The same code could 

be interpreted one way by one browser and another way by another browser. Some browsers ignore certain 

kinds of code while others will read it. The problem is made more complex by the fact that different versions of 

the same browser sometimes have different capabilities for interpreting certain kinds of code, so while Netscape 

4.7 for example, may not interpret certain code, that problem may have been resolved in Netscape 6. Things are 

complicated further by the fact that the same browser brand in Windows may not interpret the same code the 

same way as a Mac or on a PC. This discrepancy has long been a problem for designers. Vendors are interested 

in differentiating their product from others, so it is understandable that such inconsistencies would arise, but in 

another sense the situation is vexing. If each automotive company made a gas tank that took a different size 

refuelling hose, consumers would be outraged at not being able to go to any gasoline station for a re-fill. And 

yet this is the current situation on the web. 

WAI (the Web Accessibility Initiative) is a positive part of a larger universal trend to standardize certain aspects 

of web content and interface design. WAI is international in scope, spanning both government and the private 

sector (See W3: http://www.w3.org/WAI/ and IBM: http://www-3.ibm.com/able/ and Microsoft: 

http://www.microsoft.com/enable/). Its impact on interface design and consequently on the design and 

development of online educational content itself cannot be underestimated. WAI is a set of recommendations 

intended to standardize certain key aspects of interface design so that physically and cognitively challenged users 

can be on an equal footing with fully-abled users when they access information and interact with others across 

the Internet. WAI is not a law nor is it intended to compel designers to be homogeneous in their thinking. It is a 

way of encouraging us to think more carefully systematically about how we present our online content. The 

standard has been very carefully considered and collaboratively shaped. 

WAI has been steadily gaining momentum over the last five years and it is only just now taking root in the 

educational sector. In other words, the impact of this trend to standardization in general, and accessibility in 

particular has not yet been felt by the educational sector, but that day is fast approaching and there will be many 

issues with which to contend. This paper invites discussion on the kinds of preparation the educational sector 






6

will need in order to meet these new standards, and perhaps even more importantly, on the rational principles 

that might help us to decide how and where to spend resources. Of particular interest here will be the legacy 

content that has already been designed and mounted on the Internet and is currently non-compliant. 

The WAI standard has three levels of conformance, each more stringent than the preceding one. Priority one is 

what absolutely "must" be done, priority two is what "should" be done, and priority three is what "may" be done. 

There is plenty of flexibility as well: designs that cannot be made to conform need only to be offered in an 

alternative way that does conform. 

There are fourteen specific guidelines (http://www.w3.org/TR/WCAG10/) which are like first principles. These 

include things like the use of natural language and a preference for not relying on color alone as a way to 

communicate information. In order to determine how to meet these guidelines, the W3 has established 

checkpoints. Each guideline therefore has one or more checkpoints. These checkpoints 

(http://www.w3.org/TR/WCAG10/full-checklist.html) are like annotations to the guidelines and they explain 

more particularly how the guidelines can be achieved. Checkpoints in turn can be linked to specific techniques 

which usually contain the precise code that is needed to achieve the checklist item. The checklist items have 

even been extended to cover specific ways to evaluate and repair legacy interfaces that do not yet conform to 

standards (http://www.w3.org/TR/AERT) and there is a wonderful online tool known as Bobby 

(http://www.cast.org/bobby) that can also be deployed to this end. 

The 508 standard is an American alternative to the WAI (http://www.webaim.org/standards/508/checklist) 

although it may not have been intended as that. 508 applies specifically to American federal agency sites 

(http://www.cio.noaa.gov/itmanagement/508law.htm) and to all federally funded programs and services. While 

it is an emerging standard, it nevertheless differs from the WAI because it is also a federal law. The emergence 

of this new standard, while welcome in one sense, also establishes a potentially confusing alternative that has 

necessitated the W3's production of a document that maps the relationship between the two evolving standards 

and shows how and where they are similar and where they are different (http://www.w3.org/WAI/GL/508/508- 

UAAG). Canada too is now on the verge of developing its own federally defined accessibility standard and 

thereby contributing to the trend to standardization while at the same time contributing the confusion and 

complexity. 

Very few educationally-related websites (such as institutional homepages) meet even the minimum standards of 

priority one and much current online educational course content fails even more miserably. Legacy content that 

was designed in the past by any of the packaged proprietary platforms (such as WebCT, Cold Fusion, 

Dreaweaver, Front Page, Flash, Domino, Quick Place, Learning Space etc) does even not meet priority one. The 

task of re-working the design so that the interface and its code are now compliant is tedious, time-consuming, 

and often means the loss of much formatting and even some content. In most cases it is too tedious to be done 

manually so designers must wait for the vendor to come up with newer versions of these platforms so that they 

can covert the legacy content into compliance. The process of conversion can be expensive and in some cases is 

not worth it. Even more importantly, content developed in one proprietary environment is not readily 

transferable to another. In other words, let's suppose you built online content in Dreamweaver, then decided to 

make your next version of the course WAI compliant, but you wanted to convert your legacy material into 

WebCT because that compliancy making feature was cheaper than the Dreamweaver one. In this scenario, you 

could not extract your proprietary content easily if it was designed using the native code built into the platform. 

HTML files, .doc and .pdf files and media will extract fine, but things like tests, discussion forum archives and 

grades are so difficult to extract that it's often better simply to re-invent that content and then make it compliant. 

The costs and inconvenience of these vendor-forced transformations are all working against the successful 

implementation of new standards and they will prohibit institutions from switching platforms to one which is 

cheaper and or one which also complies with standards. Put another way, vendor-created dependency on its own 

proprietary platform will inevitably interfere with institutional and designer freedom to migrate to other 

platforms that are more compliant or less expensive to adopt. Thus (even new) accessibility-compliant 

proprietary platforms lock designers into platform-specific dependency that is in the always in the interest of the 

vendor, but not necessarily in the interest of the institution, the designers, the instructors, or the students. And 

while it may be "natural" for vendors to want to make their particular design tools absolutely essential for design, 

the owners of the content are also forced into a dependency on the accessibility conformance of that proprietary 

platform, and into a dependency on that company's commitment to -- and planning of -- future compliance 

standards. 

7

Of course, as educators who develop online content we are morally, ethically, and pedagogically bound take as 

many cognitive and physical disabilities into account as possible when designing and mounting content online, 

or any course for that matter. HCI (Human Computer Interaction) specialists also tell us that designing for 

disabilities has many other measurable benefits as well. For instance, an interface that is user-friendly for 

disabled people is almost always also more user-friendly for fully-abled users too. This is a win for everyone! 

There are other advantages too. Inclusive design welcomes cognitive difference and thereby helps create an 

online culture of acceptance and risk-taking, both of which are essential for learning. This kind of flexibility 

allows all kinds of nuanced differences in learning styles to thrive. This can only be a good thing. 

So here's the nub of the struggle in a nutshell. Will the moral, ethical, and pedagogical reasons in favour of 

accessibility standards inevitably be outweighed by succumbing to the platform dependence that vendors want, 

and by the prohibitive cost of re-vamping legacy content? In the next few pages I shall unpack some of these 

issues in a more detail. 

Beauty and the Beast or, the Haphazard Transformative Power of the Internet and Naïve Users 

I taught my first university class in 1978 and I first logged on to the Internet in 1986, about four years before the 

WWW made its public appearance. Although it took another few years before I was able to integrate even in a 

crude way the power of these two distinct experiences, the interval of time between 1986 and 1993 was fraught 

with the stress and anxiety that accompany dramatic growth. In addition to the unwelcome negative factors, I 

found my self-confidence rapidly eroding as my ability to learn and comprehend seemed to stall, and I noticed 

that I was all-too-frequently drawn away from issues in my own field of expertise (English Literature) and pulled 

into technical and technological problems and paradoxes. On the other hand I was able to communicate more 

frequently and conveniently with students and colleagues, and the more work I did online, the more ways I found 

to enrich the quality of the learning experience as long as there remained a continuing and strong face-to-face 

component in the instruction. It was truly a sublime experience: an unholy combination of the horrifying and the 

exhilarating. 

In the early stages of adoption and use, I naively relied on the Internet for only two purposes: social 

communication with colleagues, and locating useful information. It never actually occurred to me to use the 

Internet as a teaching tool. It became a way to communicate with colleagues in my own university at first, and in 

other places in the world later on, so in this sense, the Internet initially enriched my life in a private way. Most 

of my discoveries were accidental and serendipitous, and they occurred in the most haphazard and desultory of 

ways. I also contend that online educational content itself has developed in much the same manner, leaving a 

patchwork of different files, directories and resources each with a different set coding and formatting laws, and 

each with a different pedagogical intention. It is the haphazard nature of this evolution that is at once 

exhilarating and troublesome because very little conscious attention was paid to standards, to usability, and to 

accessibility. At a time when almost all new buildings in North American architecture recognized the need for 

wheelchair and washroom accessibility, my own online educational content seriously lagged behind in 

sensitivity to the needs of challenged users, and the presentation of my own content was the worse for it. 

Over the course of my own development I also had to overcome many formidable obstacles, such as having to 

learn many different technical protocols, technologies and the jargon of the Net, but I was also driven forward by 

my enthusiasm to initiate the "free" interaction with other human beings and to access some of the wealth of free 

information that seemed to be available out there. I learned the technical material in a "just in time" manner, just 

as I needed it. Eventually this just-in-time manner gave way to a full time obsession and I discovered I was 

spending more hours each day on studying the Internet and its web-based languages, than I was in my own area 

of expertise. The price for this indulgence was heavy: my university, like most others in North America, did not 

recognize the value of that kind of work during the 1990's, and even now still does not recognize the value of 

that kind of work unless the post-secondary degrees of the researcher are actually in that particular technological 

field. And so it is that many university faculty who once developed online in the 1990's have now withdrawn 

from the frenzy and in their fatigue and disillusionment are content to watch rather than do. In many instances, 

the expertise learned by this older generation of educators has also disappeared off the radar screen. The 

mistakes made and lessons learned by earlier designers is undoubtedly very valuable in new designs, but there 

has been an absence of genuine cooperation in the educational sector, at every level, with each faculty member 

preferring instead to create their own (ostensibly) distinctive online content from scratch or from a vendor's 

proprietary template, rather than beginning with cooperating with others. As educators we have been very 

ineffective in minimizing the cost of designing online content and very good at creating redundancies that 

duplicate sub-standard codes and sub standard features in the interfaces. 

8

In my own experience at least, it is common to see faculty and students come to the Internet with a similar kind 

of enthusiasm for interactivity and information that I myself had more than a decade ago, but the biggest 

difference between then and now is that today's world tries very hard to offer painless ways for human beings so 

to get online without having to be technically able, and herein lies one of the core problems. I see instructors 

responsible for online courses online with little (if any) knowledge of HTML, and certainly no working 

knowledge of JavaScript, Perl, XHTML and XML – and even more importantly, little or no understanding of the 

coding standards that should be implemented or why those standards might be important. Knowledge of various 

Internet protocols such as FTP and Telent is even less, and knowledge of Internet security and the myriad of 

possible ways to cheat and commit fraud is embarrassingly non-existent (this also encourages plagiarism, but 

that is a tangential topic). Instead, ready-made platforms tantalizingly offer the opportunity for a kind of stressfree 

intellectual "plug and play" whereby the instructor's "real" knowledge and expertise can be harvested and 

transformed into online content with little inconvenience. In short, ease of access and a shallow learning curve 

of 1-3 days training have been purchased at the very high price of technological ignorance and naiveté. The 

work of online course design has also become highly de-skilled by so-called easy-to-use software precisely at 

that moment in history when the complexity of the work now necessarily involves the coordination of 

instructional designers, graphics people, animators, and database programmers. All of this means that while 

online content design has become increasingly complex (In 1993 the cost of "getting in the online game" 

required technical expertise that could be learned in a few days. All I needed to know in order to put a course up 

on the WWW was some basic HTML and a little FTP. Today that technical bar is much higher: you'd need 

HTML and cascading style sheets (3 days training), JavaScript (5-10 days training), XML (2 days), XSL (2 

days), XHTML (1 day), database basics (5-10 days), graphics expertise (2-3 days), and expertise in some of the 

tools used to create courses (Dreamweaver, Cold Fusion, WebCT and so on)), the tools we use to create this 

content are naively simple ("dumbed-down") and they generate content that is incompatible with other platforms 

and that is proprietary to only a single commercial interest. 

The current world of online education that I know is also surprisingly naive about student expertise. In spite of 

frequent claims that today's student is technologically hip ("my eleven year old has more technical aptitude and 

expertise than my wife" etc etc) I note with dismay that many students do not know the difference between a file 

and a directory, nor the difference between a binary and an ASCII file: they do not know what a tree structure is, 

and even when the software platform has been idiot-proofed so that uploading a file is a matter browsing and 

clicking, they do not know how to navigate their computer in order to find a file that they have saved. I am still 

surprised when I see that my own university offers dozens and dozens of online courses and no one – student or 

faculty member – ever has to show or demonstrate any technical expertise. 

One of the (many) undesirable consequences of this increased division of labour and job deskilling is that very 

few online courses meet any of the technical or interface standards that have recently been developed by the 

World Wide Web Consortium. The educational sector may be ahead of the trend in theory and in sensitivity, but 

we lag behind in policy, budget, and ability to re-program current online resources in ways that conform to these 

new standards. 

Ian Webb has noted that the cost of accessibility-compliant design is not prohibitive if it is taken into account 

right from the onset (http://www.techdis.ac.uk/resources/webb01.html) and HCI specialists will also tell you that 

the sooner a design project commits to one particular solution (or kind of solution) the more likely it is that the 

design will be unsuccessful. Thus one of our biggest challenges will be to find ways to include and re-design 

legacy content, much of which is perfectly good material and flawed only because of its skin. Webb, however, 

does not seem to be aware of the trap of vendor specific solutions. 

A standard is a set of rules of practices that one adheres to. A protocol is a procedure, a set way of sequencing 

steps and processes. A red light means stop and a green one means go (standards). In some countries cars must 

drive on the right side of the road, and in others, on the left side (protocols). Failure to conform to rules and 

protocols on the roads will almost certainly have fatal consequences. The language that we speak is also based 

on rules (vocabulary) and protocols (grammar). The Internet too is based on standards and protocols. TCP/IP 

(Transmission Control Protocol/ Internet Protocol) is a set of rules and protocols for how information moves 

across the Internet and how unique addressing is assigned. 

The Internet is a conundrum when it comes to rules and protocols: it is an environment where many technical 

standards exist by necessity, but it is also a place where there is great resistance to cognitive and design 

standards. This resistance occurs for several reasons. First, standardization of a browser's interpretive ability to 

render HTML code in a single way seems inimically hostile to the vendor's need for product differentiation: this 

9

is an essential part of marketing in the private sector. Every browser's vendor wants its browser to do at least 

something different from the others. Thus accessibility is blocked at the design and development level by 

vendor specific (proprietary) code, and it is blocked at the client (consumption) end by the same problem with 

browsers. Please let me be clear about this. I am not blaming vendors for being self-interested. What I am 

saying, is that this inevitable and unavoidable self-interest presents an apparently insurmountable obstacle to 

standardization of any kind, especially when vendor self-interest makes cross-vendor migration so difficult and 

convoluted. As consumers, we would not tolerate a different size and thickness CD for every recording label 

that required a physically different CD player to pay it, so why would we tolerate the equivalent in our 

courseware? 

The World Wide Web Consortium has historically been as interested in process as well as product, and in the 

early days when "browser wars" were flaring up between Netscape, Mosaic and Microsoft, the W3 consistently 

urged the commercial sector to design their browsers so that they would all interpret the same HTML code the 

same way. The W3 has strived for open standards. The "nub" of the problem is that open standards are (I'm 

tempted to say always) in the interest of the consumer where as the standards developed commercial (called 

proprietary standards) are always in the interest of the commercial designers since they try very hard to 

differentiate their products from others on the market. This tension is a critical part of the problem. 

Summary 

We know that designing interfaces (and code) for accessibility is a good thing in practice and in principle, and 

therefore is desirable. By improving the quality of access for one, we improve it for all. Yet the effort involved 

in the design process occurs at a time in the Internet's evolution when the nature of the work of online 

development is highly specialized, and (ironically) where the culture also favours dumbed-down software for 

untrained programmers and users. All of this contributes to carelessness about, and ignorance of standardization. 

Furthermore, we have also inherited legacy content that does not reflect accessibility issues, and even newly 

developed accessible-compliant content is trapped in the prison of proprietary platforms that will not allow 

designers to easily migrate content from one platform to another, thus restricting freedom of choice in adopting 

other proprietary platforms that may suddenly become more fully compliant or cheaper. Finally, educational 

institutions are also trapped in fiscal constraints that provide seed money or one-time-only funding for online 

content instead of ongoing baseline budgets that allow for re-design and upgrading. 

There are many challenges to be overcome. The lure of achieving very sophisticated (non-compliant) interfaces 

with little or no programming expertise is at once liberating (democratizing) and dangerous. Funding is limited. 

Human resources are limited. Constant re-training of technical support people is essential, but expensive and has 

been in steady decline since 9/11. Under what conditions are private sector partnerships a viable solution? Can 

different institutions in the educational sector actually cooperate to the extent that they share open code and 

particular designs that can be re-used, and if so, what barriers are in the way? 

More Questions For Discussion 

Here are a few more specific questions for discussion. 

1. There is a new British government inquiry into the current state of accessibility compliance in educational 

institutions (Project Bunnyfoot (http://www.bunnyfoot.com/freestuff/articles/accessibility/access_he.html). 

Can anyone advise of a) how this study is progressing and b) are there any other comparable studies being 

done currently in other countries? Are there any hypotheses about what will be found? 

2. Apart from the fact that the 508 standard is actually a law, there seems to be significant political will on the 

part of American educational institutions to conform to the 508 standard, if not the WAI. Does anyone 

know of a) any resistance that is being expressed or b) any comparable movements in other countries? 

3. There have been enthusiastic claims that conformance to accessibility is not cost-prohibitive 

(http://www.cfilc.org/projects/cfilc_position_papers/atbriefing.html), but is there any real evidence to show 

this? A recent article in PC World (http://www.pcworld.com/resource/article/0,aid,17690,00.asp) claims 

that not even the private sector can afford the cost. Comments? 

4. Are there any cost effective software tools or organizational workflow processes currently being used to 

convert online content to open standards that are also WAI compliant? 

5. The University of Toronto's Adaptive Technology Resource Centre 

(http://www.utoronto.ca/atrc/research.html) offers an excellent webography of current research on 

10

accessibility software and tools, but will it ever be possible to implement a single open standard for online 

content (like XML) that is actually capable of allowing content creators not only to comply with 

accessibility and design standards, but which will also allow them to migrate content from one proprietary 

platform to another? What obstacles must first be overcome in order for this to happen? 

6. Use Bobby (http://www.cast.org/bobby) on your own institution or place of employment or to test your 

current online content. In your estimation, what percentage of educational (and educationally related 

content) in your country conforms to these standards? 

7. Is it better to offer full accessibility on a JIT (just in time) basis as needed when individual users appear who 

need alternate designs, or to design with accessibility in mind right from the outset? 


The most amazing thing happened when this discussion went online. On the one hand, there was almost no 

public response at all to the points raised, and on the other hand the private and largely confidential 

correspondences I received paint a very different picture from the one of apparent apathy that was manifest in 

the public forum. 

First, let's review the public forum discussion. Here the discussion was sparse and adequate, but certainly not 

controversial or innovative. On Oct 14, the following observation was made: "What happens when we get a 

situation that we're in now? A discussion about lurkers, scheduled as "future discussion", has broken out before 

the "current discussion about accessibility standards is over" (Trayner). The observation was a well-founded 

one, and one which no one attempted to address or rectify even after my interjection. The biggest exception to 

this adequate discussion was the very insightful observation on Oct 14 that these new "International standards 

could feasibly challenge the hegemony of American/English speaking presence online" (Aurilio). There is much 

to contemplate here and this is a line of inquiry that should be pursued. 

On Oct 10 there were a few more public postings that cited some web resources about accessibility, including 

http://www.uwic.ac.uk/ltsu/accessible.pdf,http://www.uwic.ac.uk/New/disability_dpet/staff/published_materials. 

asp and http://www.uwic.ac.uk/New/disability_dept/useful_resources.asp (Cooper), all of which are helpful, but 

all of which (I note in passing and with all due respect) contain a single reliance on pdf files which in and of 

themselves, are not accessibility-compliant! There was a smattering of other comments also, mostly about the 

power of Dream Weaver MX and the new WebCT, both of which are capable of producing WAI compliant code 

(Saurilio, Oct 14). 

Overall, however, the public postings were few and far between. It's often difficult to interpret silence, and this 

particular case would not have been any different had it not been for the fact that the private and confidential 

discussions that went on told a very different story. Although I have promised not to reveal particulars about 

these discussions, I can say the following things with certainty: 

1. The cost of re-developing legacy software is consistently at least twice the cost of the original development 

of non-compliant interfaces, and in many cases, is more than double. This is very troublesome. 

2. Those who are accessibility advocates are frustrated by the sluggishness of institutions to commit significant 

resources of time, money, and people, to accessibility compliance. 

3. Those who actually have to oversee and execute compliance are frustrated by the combination of 

unrealistically high costs and the fanatical zeal of compliance advocates who push for it regardless of cost 

limitations. 

4. Both those who are for and against compliance feel that the other does not understand them: neither wants to 

speak publicly about it. 

5. Reliance on private sector partnerships to achieve compliance can be successful, but can be plagued by 

problems of cost and corrupted processes. 

In short, there is a silent and deeply-rooted antagonism raging in many educational institutions over these 

compliance standards and their implications. This antagonism cannot be repaired, in my opinion, until there is 

more open and candid dialogue. 

I still don't know if the silence in the discussion forum around this issues was because subscribers did not see it 

as important (an authentic problem of ignorance), or because they think they already understand the issues (a 

problem of presumption), or if the silence is a case of Nero fiddling while Rome burns (a problem wilful 

oversight). 

In any event, I am predicting that this is a problem which, like a toothache, is going to persist until it becomes by 

necessity, a painful focal point of attention. There is much work to be done. 

11

Muirhead, B., & Juwah, C. (2004). Interactivity in computer-mediated college and university education: A recent review of 

the literature. Educational Technology & Society, 7 (1), 12-20. 

Interactivity in computer-mediated college and university education: A recent 

review of the literature 


Brent Muirhead 

Area Chair MAED Curriculum & Technology 

University of Phoenix Online, USA 

bmuirhead@email.uophx.edu 

Charles Juwah 

Senior Educational Development Officer 

The Robert Gordon University, UK 

CharlesJuwah@aol.com 


Discussion: November 10-19, 2003 

Summing-up: November 20-21, 2003 


Introduction 

Interactivity and interactions are critical in underpinning the learning process in face-to-face, campus based and 

distance and online education. Interactions serve a diverse range of functions in the educational process which 

include, for example: 

promoting active and participative learning on a one to one basis or within a group or learning community 

through social dialogue; 

enabling effective facilitation of learning to suit individual learner's needs and learning styles; 

allowing learner input to the learning process as well as enabling learners to take ownership and control of 

their learning; 

enabling the development of higher order knowledge and abilities, for example critical thinking, problem 

solving, judgement -/decision-making skills, reflection, etc.; 

providing effective feedback to inform on the teaching and learning process as well as enhance the quality 

and standards of the learning experience ( Fahy, 2003; Juwah, 2003 

The rapid evolution of the information and communications technologies (ICT) and the Internet has contributed 

significantly to the phenomenal growth of distance and online education. Educational research findings suggest 

that the success of any educational process is and should be underpinned by sound pedagogical principles and 

interactions. Our brief overview of literature will highlight some trends and developments in the study of 

interactivity and interactions in distance and online education. 

Defining Interactivity 

The search for an educationally viable definition of interactivity has produced some valuable insights for 

distance educators. Interactivity and interactions in online education are complex, multifaceted phenomenon and 

are critical in promoting and enhancing effective learning (Anderson, 2002; Hirumi, 2002; Sims, 1995; Yacci, 

2000).Yacci (2000) describes four major attributes to interactivity: 

Interactivity is a message loop; 

Instructional interactivity occurs from the student’s point of view and does not occur until a message loop 

from and back to the student has been completed; 

Instructional interactivity has two distinct classes of outputs: content learning and affective benefits; 

Messages in an interaction must be mutually coherent (p. 6). 

Yacci’s reflections reveal the existence of a student-centered orientation around their perceptions of interactivity. 

Therefore, a web based educational program can claim interactivity but students will not acknowledge 

interaction until they individually receive some form of feedback. Yacci’s observations emphasis the need to 






12

study online interaction from a communication theory perspective by investigating a diversity of variables such 

as length and number of messages, type of information shared and the amount of time between responses. 

Muirhead (2000) offers a practical definition of interactivity, which affirms the human dimension of this term; 

interactivity refers to communication, participation, and feedback. Additionally, interactivity involves 

participation by the learner in online communication with other learners and with their instructors. The definition 

highlights the personal nature of sharing information during an online class. Naturally, students interact with 

their course materials through reading textbooks, journals and discussion forum comments from other students 

and their instructors. The subject matter provides an academic foundation for meaningful dialogue within a 

distance education class. 

From the above definitions, it is clear that interactivity is a multifaceted concept and can be described to mean 

different things in a variety of contexts. Nevertheless, it is recognised as an important and critical characteristic 

in instructional design, social context and success of distance education (Beard and Harper, 2002). Thurmond 

(2003) shares an insightful definition of interaction: 

The learner’s engagement with the course content, other learners, the instructor, and the technological 

medium used in the course. True interactions with other learners, the instructor, and the technology results in a 

reciprocal exchange of information. The exchange of information is intended to enhance knowledge development 

in the learning environment. Depending on the nature of the course content, the reciprocal exchange may be 

absent – such as in the case of paper printed content. Ultimately, the goal of interaction is to increase 

understanding of the course content or mastery of defined goals (p. 4). 

To add to this debate, the authors based on their understanding and experience from practice share relevant 

definitions of interactivity and interaction as follows: 

Interactivity 

Interactivity in distance and online education describes the form, function and impact of interactions in teaching 

and learning. 

Interaction 

Interaction is a dialogue or discourse or event between two or more participants and objects which occurs 

synchronously and/or asynchronously mediated by response or feedback and interfaced by technology. The 

interactions which can be categorised as learner to learner, learner to content, learner to tutor, learner to 

technology, tutor to content, tutor to technology, content to content, promote and enhance quality of active, 

participative learning in a learning environment (See figure 1). 

Types of Interactions 

There exists in the literature an array of taxonomies for categorising interactions. Several authors have identified 

four primary types of interactions. These are - student-student, student-teacher, student-content, studentinterface 

(Anderson 2002; Hirumi, 2002; Rovai, 2002; Sims, 1995). Sims (1995) provides a valuable 

classification of interactivity based on an instructional courseware designer’s perspective. This classification 

demonstrates both the importance and integrated aspects of the various concepts in enhancing motivation, 

engagement and instructional transactions in technology-enhanced education. The following descriptions depict 

the range and characteristics of the interactive concepts: 

Object Interactivity: (proactive inquiry) refers to an application in which objects (buttons, people, and things) 

are activated by using a mouse or other pointing device to elicit an audio-visual response. 

Linear Interactivity: (reactive pacing) refers to applications in which the user moves through predetermined 

linear sequence of instructional material without any response-specific feedback to learner’s actions. This type of 

interaction is referred to as “page-turning”. 

13

Socialising 

Exposition 

Questioning 

Dialogue 

Responding 

Paraphrasing 

Prompting 

Reinforcement 

Directing 

Encouraging 

Reviewing 

Reflection 

Tutor/Facilitator 

Student/Learner 

(One to one 

interaction) 

Interface 

: 

(technology) 

Virtual 

Simulation 

Group of 

Learners 

Clicking 

Browsing 

Manipulating 

objects 


Clicking 

Browsing 

Manipulating 

objects 


Browsing Socialising Drill and Practice 

Problem solving Participation Responding 

Analysing Questioning Dialogue 

Reviewing Reflection 

Figure 1. Model of Interactivity 

Content 

Updating 

Scheduling 

Managing learner’s progress 

Simulating 

Support Interactivity: (reactive inquiry) provides learners with performance support in both generalised and 

context-sensitive perspectives. 

Update Interactivity: (proactive) relates to individual application components or events in which a dialogue is 

initiated between the learner and computer-generated content. This may involve applications which present or 

generate problems or dialogues to which the learner must respond. The learner’s response will result in a 

computer-generated update or feedback. The instructional rigour of the judging will determine the extent to 

which the update or feedback provides a meaningful response to the user. 

Construct Interactivity: (proactive elaboration) involves learner in manipulating component objects to achieve 

specific goals and/or outcomes. This type of interaction provides a link between non-situated learning and 

simulated environments, and introduces the learner to authentic learning situations without the risks or costs 

involved with “real life situations”. 

Reflective Interactivity (proactive elaboration) refers to interactions in which users’ entered responses to a task 

are compared to the responses of other users as well as recognized "experts", thus enabling the learners to reflect 

on their response and make their own judgment as to its accuracy or correctness. 

Simulation Interactivity:(which ranges from reactive elaboration to mutual elaboration, depending on its 

complexity) involves the learner in manipulating “non-real” objects to obtain desired goals in a training 

sequence. Sims (1995) posits that simulation and construct interactivity levels are closely linked, and may 

require the learner to complete a specific sequence of tasks before a suitable update can be generated. The 

interaction sequence can also be varied for example, allowing the learner to progress to other stages of 

learning/activity only after making a correct choice. 

Hyperlinked Interactivity: (proactive navigation) provides the learner access to a wealth and diverse range of 

information linked to a knowledge base. 

14

Non-Immersive Contextual Interactivity: (mutual elaboration) provides the virtual environment in which 

users/learners engage in meaningful learning in a job-related context through a series of content oriented 

sequences. 

Immersive Virtual Interactivity: (mutual elaboration) provides a complete computer-generated, virtual reality 

interactive environment in learning based on interactions between the user’s actions and response and feedback 

from within the learning environment. 

He concluded his classification by proposing an engagement-control model of interactivity. The model consists 

of engagement which is instructional or navigational, control wherein the program or learners is in control of 

making the instructional/navigational decisions and the interactive concept provides an indication of the type of 

interaction expected under the particular context. 

Hirumi (2002) provides a concise summary of the categorisation of interactions as“communications-based”; 

“social”; “roles of the instructor”, “purpose-based”; “use of telecommunication tools”and“activity-based”. In 

addition, Hirumi (2002) highlights the importance of sound educational principles, cognitive learning theories 

and grounded instructional strategies to inform course design and sequencing of activities to ensure effective 

interactions, thereby, making learning relevant, meaningful and authentic. 

However, to support authentic learning as well as enhance the learner’s educational experience in distance and 

online courses, it is imperative to provide adequate scaffolding. The epistemological approach in providing 

appropriate scaffolding to support deep learning in the diverse range of interactions can be via – manipulating 

objects and symbols, questioning, dialoguing, analysing, netweaving, representing, i.e. presenting and 

structuring activities and guiding learner’s reflection within appropriate contexts. These scaffoldings can be 

categorised as: 

Conceptual: These guide the learner in what to consider, particularly when the problem/task is defined. They 

provide explicit hints and examples. 

Metacognition: These guide the learner on how to think in considering the problem/strategies, for example, 

framing the problem. These provide suggestions to plan ahead, model cognitive strategies, regulatory process 

and evaluation. 

-Procedural: These guide the learner on how to utilise information – i.e. provide on-going help and advice, and 

may include tutoring. 

Strategic: These guide the learner in analysing and approaching the problem with a strategy. These provide a 

start up to seeking solutions, as well as enabling focused responses to the problem situation (Juwah, 2002). 

Cognisant of the role of interactions in education and drawing from experience and other research studies, 

Anderson (2002, paragraph 10) goes on to develop an “Equivalency theorem” that states: 

Sufficient levels of deep and meaningful learning can be developed as long as one of the three forms of 

interaction (student-teacher; student-student; student-content) are at very high levels. The other two may be 

offered at minimal levels or even eliminated without degrading the educational experience. High levels of more 

than one of these three models will likely deliver a more satisfying educational experience, though these 

experiences may not be as cost or time effective as less interactive learning sequences. 

Our search and review of the literature highlights six primary types of interactions within which a variety of 

secondary interactions and activities are embedded. These categories are: 

student-student; 

student-teacher; 

student-content; 

student-interface; 

teacher-teacher; 

content-content. 

Notwithstanding the plethora of categorizations of interactions, one thing was obviously clear in the literature. 

There is no single medium that is superior to the others in supporting the learners’s needs and their educational 

experience via the provision of various types of interactions. However, each type of instructional interaction 

15

plays a role in the entire educational process, with the process being more effective if predicated on a blend of 

interactions. 

Educational Implications 

Research has shown that the use of ICT and multimedia in both verbal and non-verbal forms improve and 

facilitate learning through reducing cognitive load. It provides the right context and an integrated learning 

environment that combines the use of the Web and an appropriate mix of multiple - and/or multi-media e.g. 

animation, audio, images, video, CD, print and hypertext to give a rich, stimulating and interactive learning 

environment. The media mix enhances learner motivation and has the potential of meeting the needs of the 

different learning styles – visual (images), auditory (sound), tactile (touch) and kinaesthetic (whole being). 

However, it is critical that in designing a learning environment in which ICT is used to support learning that such 

an environment has the ability to synchronise and coordinate diverse multimedia elements (Juwah, 2002). 

Research studies on constructivism and interactivity point to some interesting preliminary results. Taylor and 

Maor (2000) studied a graduate online class at Curtin University of Technology, Perth, Australia. The research 

project created a questionnaire known as the Constructivist On-Line Learning Survey (COLLES) to measure 

both teacher and student perceptions in the following six categories: 

professional relevance- the extent to which engagement in the on-line classroom environment is relevant to 

students’s professional worldviews and related practices; 

reflective thinking- the extent to which critical reflective thinking is occurring in association with online 

peer discussion; 

interactivity-the extent to which communicative interactivity is occurring on-line between students and 

between students and tutors; 

cognitive demand- the extent to which communicative interactivity is occurring on-line between students 

and tutors; 

affective support- the extent to which sensitive and encouraging support is provided by tutors; 

interpretation of meaning- the extent to which students and tutor co-construct meaning in a congruent and 

connected manner (Taylor and Maor, 2000, paragraph 4). 

Student expectations were met in five of the six categories except in the area of interactivity. A revealing finding 

was the absence of dynamic dialogue in the class which had structured small group activities that included a 

systematic change of student leaders and topics. Student online remarks were one-dimensional commentaries 

that failed to address comments made by their colleagues. The study indicated teachers must create a learning 

environment that equips students with instructional experiences to enhance their reflective skills. Additionally, 

students must be dedicated to becoming more sophisticated learners who are willing to learn from their 

colleagues while cultivating an intellectually engaging writing style that fosters academic discussion. 

It is clearly evident from the literature that interactions are critical for enhancing motivation, communication, a 

diverse range of skills and intellectual development in the educational process. However, the lack of proper 

integration between pedagogy, organization and technology has often resulted in some distance and online 

education being delivered as correspondence courses, with the consequence that such courses lack interactivity, 

immediacy and appropriate tutor feedback. Such a phenomenon has led Garrison and Anderson (2003) to state 

“educators have not understood and capitalized on the blend of symbol systems, such as multimedia, text-based 

communication systems that create new modes of expression and communication” (p. 4). 

Further Research 

Information available in the literature on research into the complex phenomenon of interactivity and interactions 

is rather limited in scope due to the lack of theory to guide research projects (Anglin & Morrison, 2003). Berge 

and Mrozowski (2001) in their survey of research articles from four technology journals for the period 1990- 

1999, identified the following research trends: 

Most attention-over 100 articles were focused in three categories 

design issues 

strategies to increase interactivity 

learner characteristics 

16

Least attention was paid to 

learner support 

equity and accessibility 

cost/benefit trade-offs 

Interactivity has been a major focus for researchers but much more needs to be done. Interactions online occur 

within a learning community and such communities provide an important area for research, in terms of the 

nature of collaboration and interactions within the community of learning. The issue of learner support is 

connected to related topics such as student attrition. For instance, what are the most effective types of learner 

support? Motivation and engagement are critical factors for effective learning. The challenge here is to 

investigate the pedagogy of engagement and interactions through electronic simulation or virtual reality in 

enhancing learner’s experience. 

Conclusion 

This literature review highlights the multifaceted nature of the concept of interactivity and interactions, as well 

as the importance of interactions in underpinning distance and online education. It briefly highlights the fact that 

interactions are not solely the manipulation of symbols and representation but the promotion of metacognition 

(reflection) which is critical in meaning making and construction of new knowledge. Additionally, the review 

also highlights insights from online teaching experiences that will help inform current theories and generate 

ideas to develop new theories (Anglin & Morrison, 2003). 

Discussion Questions 

1. What types of interactions provide the best educational experiences for online students? 

2. What are the most effective ways to facilitate student collaboration online? 

3. What teacher practices encourage positive communication within the online class? 

References 

Anderson, T. (2002). An Updated and Theoretical Rationale for Interaction. Available: 

http://it.coe.uga.edu/itforum/paper63/paper63.htm. 

Anglin, G. J. & Morrison, G. R. (2003). Evaluation and research in distance education: Implications for research. 

In C. Vrasidas & G. V. Glass (Eds.). Distance education and distributed learning, (pp. 157-180). Greenwich, Ct: 

Information Age Publishing. 

Beard, L. A. & Harper, C. (2002). Student perceptions of online versus on campus instruction. Education, 122, 

658-663. 

Berge, Z. L. & Mrozowski, S. (2001). Review of research in distance education, 1990-1999. The American 

Journal of Distance Education, 15 (3), 5-19. 

Fahy, P. J. (2003). Indicators of support in online interaction. International Review of Research in Open and 

Distance Learning. Available: http://www.irrodl.org/content/v4.1/fahy.html 

Garrison, D. R. & Anderson, T. (2003). E-learning in the 21 st century: A framework for research and practice. 

London, UK: RoutledgeFarmer. 

Hirumi, A. (2002), The Design and Sequencing of e-Learning Interactions: A Grounded Approach, International 

Journal of E-Learning, Vol. 1, pp.19-27. 

Juwah, C. (2003). Using Peer Assessment to Develop Skills and Capabilities. Journal of the US Distance 

Learning Association – Available: http://www.usdla.org/html/journal/JAN03_Issue/article04.html 

Juwah, C. I. (2002). Using Information and Communication Technology to Support Problem Based Learning. A 

commissioned article by the Institute for Learning and Teaching in Higher Education (ILTHE). ILTHE Members 

17

Resource Area. [Access is restricted to members only] Available: 

https://www.ilt.ac.uk/portal/showarticle.asp?_article=3581 

Mayes, T. (2000). Pedagogy, Lifelong Learning and ICT. A Discussion Paper for the IBM Chair presentation. 

http://www.ipm.ucl.ac.be/ChaireIBM/Mayes.pdf 

Muirhead, B. (2000). Interactivity in a graduate distance education school. Educational Technology & Society, 3 

(1), 2000. Available: http://ifets.ieee.org/periodical/vol12000/muirhead.html 

Rovai, A. A. (2002). A preliminary look at the structural differences of higher education classroom communities 

in traditional and ALN courses. Journal of Asynchronous Learning Networks, 6 (1). Available: 

http://www.aln.org/alnweb/journal/jaln-vol6yissue1.htm 

Sims, R. (1995). Interactivity: A Forgotten Art? Instructional Technology Research Online. Available: 

http://www.gsu.edu/~wwwitr/docs/interact/ 

Taylor, P. & Maor, D. (2000). Assessing the efficacy of online teaching with the Constructivist On-Line 

Learning Environment Survey. In A. Herrmann & M. M. Kulski (Eds.), Flexible futures in tertiary teaching. 

Proceedings of the 9 th annual teaching learning forum, 2-4 February 2000. Perth, Australia: Curtin University of 

Technology. Available: http://cea.curtin.edu.au/tlf/tlf2000/taylor.html 

Thurmond, V. A. (2003). Examination of interaction variables as predictors of students’ satisfaction and 

willingness to enroll in future Web-based courses. Doctoral dissertation. University of Kansas Medical Center, 

Kansas City, KS. 

Yacci, M. (2000). Interactivity demystified: A structural definition for distance education and intelligent 

computer-based instruction. Educational Technology, XL (4), 5-16. 


The discussion began with concerns about creating an accurate definition of interactivity that clearly describes 

the human interactions in the online environment. Writers have developed several taxonomies of online 

interactions but it is still a work in progress. The dialog did engage in an assortment of interactivity issues such 

as relevant pedagogical activities, teacher competencies, teacher training, instructional design considerations and 

classroom management. 

Research studies on interactivity reveal a multidimensional entity that often requires more investigation. The 

following IFETS discussion highlights as well as reflects a rich diversity of thought on interactivity in computermediated 

classes. 

Marshal Anderson- raised concerns that instructional designers must work harder at developing e-learning 

platforms that meet legitimate student needs. 

Richard Dillman- discussed reasons why students drop out of their online classes and observed that it takes 

time for online teachers to adopt and modify their instructional techniques. "Just as it requires some new skills to 

study online, so does it require new teaching skills? If we were to be honest about it, we would have to admit 

that some teachers still have some work to do in that regard." 

Anita Pincas- shared her work as an online instructor at the University of London and how teachers can transfer 

traditional methods into the online environment by using video tapes/cameras and Power Point slides. Anita 

notes that her "Replication Model" has helped students and educators to make a natural transition to the online 

setting. 

Hans Horwath- argues for creating unique online courses that are truly different from the traditional face-toface 

classes and suggests creating an international forum as one way to bring diversity into cyber classes. 

Barry Porter-stressed the importance of student motivation and having his online masters degree accepted for a 

doctoral program. 

18

Bob Valiant- noted how individuals are already managing their own learning more than 

we probably realize and shared a web site for one of his articles on this vital subject. 

Hai Zhang- outlined potential ways to discuss the transfer of information online such as data mining, cultural 

analysis, media communication and cognitive science. 

Marsha Hammond- shared the importance of instructors communicating clearly in their online courses through 

written messages. Marsha spoke about variability of student involvement in online discussions and the desire to 

discuss this issue. 

Wang Xiuwen- stressed the need to individualize student assessments and the importance of recognizing student 

learning styles in the teaching and learning process. 

M. Yasar Ozden-obvserved that his online class requires much more work than his traditional classes. The 

human and social dimension of learning is very important to the online learning process which affirms the need 

for diligent and trained instructors. He raised thoughtful questions about the daily online management strategies 

of teachers and whether online teacher competencies differ from the traditional teacher skills. 

Ania Lian-suggested that those concerned with the constructivist perspectives of online teaching should focus 

more attention on how their instructional practices influence their student's ability to form or build 

reality/knowledge. Ania raised a vital question, "what makes our definition of interaction challenge our teaching 

rather than being subservient to our prejudices?" 

Anthony Trippe- noted how he fosters active participation in his online classes through his grading procedures 

which place a strong value on student participation. 

Mark Nichols-raised questions about the nature of interactivity and proposed that the concepts of "object" and 

"linear" should be classified as types of content navigation and not interactivity. 

Bronwyn Hegarty- explored the five different threads in our IFETS interactivity discussion to examine the 

coherence of our dialog. 

Eshaa M. Alkhalifa- raised an assortment of reflective questions about online assessment involving the teaching 

and learning process and shared insights from researchers on several issues. 

Mary Hall- shared a detailed overview of our discussion to highlight the temporal nature of interactions while 

stressing the coherence and purposefulness of our dialog. 

Roger Hartley- offered insights into assisting online learners by developing specific student plans based on 

instructors and students collaborating to create relevant course work. Roger encourages his students to map out 

their current level of knowledge which helps to identify their genuine learning needs. 

Charles Adamson- noted the role of assessment in his Japanese nursing degree program and that the method of 

assessment must affirm program goals. 

Joanna Howard- related how working in an MBA program, collaborative mapping between student and 

instructors helped accelerate the entire learning process. 

Alfred Bork- emphasized the issues of frequency and quality of online interactions. He suggested the key to 

increasing individualized education was creatively using the computer as an adaptive tool to communicate at a 

global level. He noted that adaptive tutorial learning could be quite useful for China and it has larger 

implications due to the shortage of teachers. 

Ramesh Sharma- shared recent research studies on interactivity which revealed how online instructors brought 

a human dimension to their classes to create cyber communities. 

Eric Flescher- related a concern about the weakness in current Internet based instructional activities that do not 

encourage students to do original and reflective work . "Teachers many times either make the activities too 

structured or too flexible and fail to use various multiple intelligence mode based activities (drawing, group, 

visual, mathematical etc) to supplement their activities." 

19

Bill Williams- addressed a host of interactivity issues involving creating an appropriate and stronger 

interactivity definition which avoids excessive generalizations and simplistic identification of basic interactions 

(i.e. reading), importance of properly assessing online learning, solitary learners represent a smaller portion of 

students who have few interaction expectations and critical multimedia and task design. He made a very relevant 

observation that "I believe course design and more specifically task design is a key factor here and I am 

concerned that this meta-structural aspect may be in danger of being neglected if our attention is focused mainly 

at the interaction level." 

Wang Xiuwen- related the importance of online instructors providing adequate guidance for their students when 

using a student-centered instructional model. It takes time for teachers to change from traditional teaching 

methods to technology oriented learning strategies. 

M. Goswamy- stressed the need to develop an educational online environment that promotes a sense of 

"openness" by stimulating student questioning and inquiry during online interactions. "It is, therefore, essential 

that the contents and the interaction are not only linked and complimentary but also supplementary in nature." 

Bob Cavenagh- observed that culture can play a role in online interactivity. "I deal with some students from 

other cultures and have gradually recognized that their occasional unwillingness to participate in some situations 

stems from cultural values, not from shyness, linguistic shortcomings, or lack of ability." 

Terry Anderson - shared the idea of "equivalency theorem" which provides a rationale for concentrating on 

learner needs for different types of interaction (time and geographic constraints etc.) and the cost implications of 

each type. 

The diverse range of factors covered in the discussion reflected the role and importance of interactivity in 

underpinning and enhancing learners’ educational experience. The issues raised in the discussion thus focuses 

the mind as well as challenges educators and researchers to engage more with and to continually research and 

gain a better understanding of this critical phenomenon that can impact very significantly on the quality of the 

learning process. To that end, the areas highlighted from our discourse for further research include: 

Designing of learning environments to support quality interactions; 

Course design to ensure appropriate content and effective sequencing of activities to promote quality 

interactions and learning to meet individual needs; 

The role and impact of media on interactivity and cognitive load; 

Teacher competencies and roles in enhancing interactivity in the learning environment whilst paying 

particular attention to individual student’s needs; 

Using assessment to underpin and promote effective interactions in learning situations; 

Impact of culture and/or cultural factors in classroom interactions and learning; 

Cost benefit analysis of designing interactivity into courses whilst maintaining quality learning. 

20

Barojas, J. (2004). Teacher Training as Collaborative Problem Solving. Educational Technology & Society, 7 (1), 21-28. 

Teacher Training as Collaborative Problem Solving 

Jorge Barojas 

Unidad de Telemática para la Educación 

Centro de Ciencias Aplicadas y Desarrollo Tecnológico (CCADET), UNAM 

Circuito Exterior, Cd. Universitaria. A.P. 70-186 

C.P. 04510. México, D.F., México 

Tel: (525)-5622-8602, ext. 102 

Fax: (525)-5622-8650 

jbw40104@servidor.unam.mx 

Abstract 

A problem solving protocol is described and then applied to a teacher training problematic situation in 

science and technology education. The problem consists of improving a learning community involved in 

education research, development and communication. Some remarks concerning the strengths and 

limitations of using the protocol are made in connection with action learning and collaborative work. 

Keywords 

Action learning, Collaborative work, Human learning systems, Learning community, Problem solving analysis 

protocol 

Introduction 

Systems can be classified in different ways; for instance, Wilson (1990) considers the following four categories: 

natural or physical, artificially designed, for human activities, and socio-cultural. For simplicity, we will refer to 

the first and second ones as physical systems and to the third and fourth ones as human learning systems. In 

physical systems the components and interactions are described with relatively high precision and wellorganized 

structures; very often optimized conditions for their most efficient performance can be calculated. 

Usually, for these problems the statements are quite clear, the parameters defining the system are known or can 

be determined with good precision, and we can be rather sure when the solutions have been obtained. 

Human learning systems are defined by sets of activities concerning the planning, development and evaluation of 

different sorts of transformations in human organizations where learning is at the core. For instance, systems 

concerning education, training, production, planning and management belong to this category. Practical 

expectations to improve the functioning of these systems by solving specific problems relate at the most to 

coarse descriptions and partial evaluations of how the systems work. In such cases, the problems are not well 

defined and the contexts change a lot; furthermore, there never is certainty that any solution is the optimum one 

but just the best possible one under certain given conditions. The aim is to understand concrete problematic 

situations with the hope of transforming them. 

In this report we circumscribe to educational situations and consider teacher training as a problem in a specific 

human learning system of importance in science and technology education. We consider the use of a problem 

solving protocol to address the following problem: How can we improve a learning community of educators 

working in teacher training? In what follows we describe the cognitive and metacognitive dimensions of a 

problem solving protocol called TADIR, then we present an example of application of this protocol in order to 

get first and second order solutions of the previous problem and close with some considerations regarding action 

learning and collaborative work. 

Description of a Problem Solving Protocol 

The name TADIR of the proposed problem solving protocol corresponds to the five initials of its steps (Barojas 

& Perez, 2001). While the first four steps (T-Translation, A-Analysis, D-Design and I-Implementation) 

correspond to the cognitive dimension of the protocol and lead to model building, the fifth step (R-Review) 

represents the metacognitive dimension of the protocol and provides an evaluation of the solution of the 

problem. In defining these steps we took into account the procedure introduced by Checkland (1981) and 

presented by Wilson (1990) in the treatment of human learning systems. We shall apply this problem solving 

protocol to a particular human learning system: a learning community of educators working in teacher training 

projects. 






21

We will understand by a learning community (LC) any group of human beings in interaction with the four fold 

purpose of being informed, organize communications, obtain and apply knowledge, and make transformations 

possible in order to learn something. Any LC is defined in terms of the transformation activities in which their 

members are involved in order to accomplish certain goals. Models describing the building process of any LC 

require both a conceptual framework and a logistic scenario. The theoretical considerations that serve to 

understand the beliefs, ideals, concepts, attitudes and values of the members of the community provide its 

conceptual framework. The working conditions of the community and its operational principles define its logistic 

scenario; it concerns the available technological support and the practical skills required to function under these 

circumstances. 

In this section we describe the five steps of the protocol. In order to be more concrete, we focus on the 

characteristics of one LC of trainers, not on their training projects. 

TRANSLATION (T): description of the elements of the system. 

Explain the structure of the human learning system under consideration and answer the following questions: who 

belong to the system?, in what transformation activities are they involved?, for what purposes?, in what subject 

matters?, and with what resources? 

ANALYSIS (A): characterization of the working conditions of the system. 

Make explicit descriptions of the main factors explaining the functioning of the system: objectives, restrictions, 

and connectivity. 

DESIGN (D): conceptual model containing the elements of the solution. 

Propose a graphical representation of the interacting elements encompassing the cognitive space of the intended 

solution. This diagram represents a first order conceptual model of the solution in the human learning system 

under consideration: our LC is described in terms of three transformation activities (education research, 

development and communication) and their corresponding domains of action are depicted inside dotted boxes in 

Fig. 1. At this point there is no attempt to get a complete solution to the problem, only to consider a broad 

definition of the problematic situation. 

TEACHING ACTING 

EDUCATION RESEARCH LC DEVELOPMENT 

LEARNING 

COMMUNICATION 

PRODUCTIONS SERVICES PARTICIPATIONS 

Figure 1. First order conceptual model defining a learning community (LC) 

REFLECTING 

22

IMPLEMENTATION (I): application of monitoring and control mechanisms. 

The monitoring and control of the fulfillment of the transformation activities serve to prove that the first order 

conceptual model leads to a preliminary solution, which must provide a first description of our LC and indicate 

where there are opportunities for its improvement. The first aspect is of structural nature and it corresponds to a 

mechanism monitoring the degree of maturity shown in the transformation activities. The second aspect is of 

functional nature and it is revealed through a control mechanism determining the performance of the domains of 

action defining the transformation activities. 

The monitoring process is defined in terms of the “21-st- Century Skills” which correspond to digital-age 

literacy, inventive thinking, effective communication, and high productivity (NCREL, 2000). In the case of our 

LC, the working conditions of the community can be related to the following moments (Barojas et al., 2001): 

M1: Induction: use of traditional technology like chalk and board in the style of conventional classroom 

settings; however, teaching and learning are critically reviewed by considering the opportunities provided by 

new technologies. 

M2: Transition: familiarity with computational technologies in more modern classroom settings where 

teaching and learning starts to be more effective. 

M3: Mastery: applications of information and communication technologies (ICT) guided by sound and 

realistic pedagogical principles and procedures enable the creation and improvement of virtual environments 

serving as empowerment tools for all our LC members. 

The control of the activities is made by comparing the performance of the members of the community with 

respect to four pragmatic pedagogical principles governing science education, taken from Linn and Hsi (2000) 

and summarized in Table I: 

PRINCIPLES COMPONENTS 

Encourage students to build on their scientific ideas as they develop more 

P1 

Making science accessible 

P2 

Making thinking visible 

P3 

Helping students learn from 

each other 

P4 

Promote lifelong science 

learning 

powerful and useful pragmatic scientific views. 

Encourage students to personally investigate relevant problems and revisit their 

science ideas regularly. 

Scaffold science activities so students participate in the inquiry process. 

Model the scientific process of considering alternative explanations and 

diagnosing mistakes. 

Scaffold students to explain their ideas. 

Provide multiple, visual representations from varied media. 

Encourage students to listen and learn from each other. 

Design social activities to promote productive and respectful interactions. 

Scaffold groups to design criteria and standards. 

Employ multiple social activity structures. 

Engage students in reflecting on their own scientific ideas and on their own 

progress in understanding science. 

Engage students as critics of diverse scientific information. 

Engage students in varied, sustained science project experiences. 

Establish a generalizable inquiry process suitable for diverse science projects. 

Table 1. Pragmatic pedagogical principles for scaffolded knowledge integration. 

REVIEW (R): reconsideration of the previous four TADI steps. 

The last step of TADIR examines results and procedures in order to review both our understanding of the 

conceptual model and the practical implementation of the solution. This step corresponds to metacognition, 

which according to Schoenfeld (1992) “it is knowledge and control on cognition”. In our case metacognition 

means going back to the statement of the problem through successive reconsiderations of the complete solution 

process and implies working on higher order conceptual models of our LC. A second order model is obtained by 

considering each transformation activity as an individual subsystem with meaning and purposes (see Figures 2, 3 

and 4, where a dotted box gives the meaning and a double framed box represents actions or products.) 

23

EDUCATION 

RESEARCH 

ACTING 

DEVELOPMENT 

REFLECTING 

TEACHING 

LEARNING 

• PLAN 

• BUILD KNOWLEDGE 

• EVALUATE 

• MAINTAIN 

UNDERSTAND CRITICAL FACTORS AND 

OPERATIONAL PRINCIPLES GOVERNING LC 

• ACQUIRE KNOWLEDGE 

• DEVELOP SKILLS 

• IMPROVE ATTITUDES 

• SOLVE PROBLEMS 

• WORK ON PROJECTS 

Figure 2. Second order conceptual model for education research 

• SUPPORT ARGUMENTS 

• PROVIDE EVIDENCE 

• ACCOMPLISH GOALS 

• REDEFINE TASKS 

MAKE THINGS WORK ACCORDING TO PLANS AND 

DETERMINE DEGREE OF QUALITY AND SUCCESS 

• DECIDE 

• SUPERVISE 

• CONTROL 

• JUDGE 

Figure 3. Second order conceptual model for development 

COMMUNICATION 

SEND AND INTERPRET 

DIFFERENT MESSAGES 

PRODUCTIONS SERVICES 

PARTICIPATIONS 

• LECTURE NOTES 

• PAPERS 

• MANUALS 

• REPORTS 

• MULTIMEDIA 

• TRAINING 

• CONSULTING 

Figure 4. Second order conceptual model for communication 

• LECTURES 

• WORKSHOPS 

• SEMINARS 

• CONFERENCES 

24

Application of the TADIR Problem Solving Protocol 

We are interested in a particular LC in charge of the organization and development of two different teacher 

training projects referred to the following populations: dentists teaching a laboratory course at college level and 

high school physics teachers. In both cases the training was related to ICT: how, when and for what purposes 

resources such as e-mails, electronic forums, web pages or videoconferences would be used (Bates, 1995; 

Jonassen et al., 1999; Forcier,1999). 

Without going into the detailed process of applying the first order conceptual model to the proposed problem 

(the improvement of the LC of trainers), in what follows we report step by step on the application of the protocol 

up to a second order. 

Review of the Translation Step 

The questions answered in the first step of TADIR are summarized below. A full report describing the two 

training projects will be published elsewhere. 

Who belongs to it? The nine members of this LC have different experiences as teacher trainers in science and 

technology education. Their professional backgrounds correspond to three engineers, four physicists, a designer, 

and a dentist; three of them are women and six are men. 

In what transformation activities are they involved? Although there is no sharp distinction, three members 

mostly work on education research projects, four members on development, and two members on both. All the 

members of LC are active in different communication activities and eight of them teach in high school or 

college. 

For what purposes? While members involved in education research pursue postgraduate studies (three at 

Doctoral level and one at Master´s level), members involved in development work participate on educational 

projects in their own institutions. 

In what subject matters? The activities under consideration, which determine the conceptual space where the 

solution of the problem is worked out, relate to problem solving, metacognition, teacher training, collaborative 

work, action learning, leadership in education, applications of ICT in education, design of educational games, 

and distance education. 

With what resources? Available resources include the infrastructure of the working places of the members of 

our LC, their salaries, and a grant from the National University concerning fellowships and computer equipment. 

Review of the Analysis Step 

According to the TADIR protocol, this step involves the definition of three elements: 

Objectives: O1 - to find out the critical factors defining the creation, development and consolidation of the two 

teacher training projects, and O2 - to establish the operational criteria required for evaluating and improving the 

solution of the problem. 

Restrictions: These are human interaction factors modulating the performance of the members of LC, for 

instance: appropriate use of infrastructure, experience using ICT for educational purposes, degree of 

identification with their teacher training project, quality concerning collaborative work, criteria for decisionmaking… 

Connectivity: The instrument of interaction of the members of our LC was an electronic forum 

(www.alexandria21.net) serving to foster communications, discussions, and collaborative works. 

By taking into account these three elements and by comparing the progress made during two years, we detected 

places where adjustments and changes were imperative for the trainers and of consequence for the trainees. 

Some of these issues referred to the distribution of contributions and responsibilities in each training project, as 

well as concerns with group integration and negotiation procedures (Baker, 2002). 

25

Review of the Design Step 

The three subsystems composing the second order conceptual model, previously described in Figures 2, 3 and 4, 

are by themselves complex and dynamic. They are not enough to represent a complete working definition of the 

solution of the problem; they just contain more information which leads to a better approach to the solution. 

In these figures the double framed boxes represent actions or products that have not been described, also there is 

no indication of the procedures for accomplishing them nor the instruments required for their evaluation. A 

report that will include these aspects will correspond to a third order conceptual model. Up to now we just 

intended to show how the application of the problem solving protocol generates a constructive process of 

thinking and acting to solve problems in human learning systems. 

Review of the Implementation Step 

The review of this step implies an evaluation of the level of performance of the members of our LC made by 

applying the mechanisms of monitoring and control previously described. An example of such an application is 

given in Table 2 where the first column indicates the transformation activities of the first order conceptual model 

(ACTIVITIES), the second column represents their domains of action (DOMAINS) and the next seven ones 

correspond to the following aspects: the three moments referring to the use of ICT (M1 – Induction, M2 – 

Transition, and M3 – Mastery), and the four pragmatic pedagogical principles described in Table 1, indicating the 

performances of the members of the community (P1 – Science accessible, P2 – Thinking visible, P3 – 

Collaborative learning, and P4 – Lifelong learning). 

For the evaluation of the performance of our LC we have used the following graded scale for which the 

corresponding numerical values are given in parentheses: inexistence (0), existence (1), sufficiency (2), 

appropriateness (3), mastery (4), and effectiveness (5) (Wiggins & McTighe, 1998). These values reflect in 

average a recent situation of our LC after two years of work, showing places where improvements are required. 

Here we do not pursue the interpretation of the implications of these indicators. 

ACTIVITIES DOMAINS M1 M2 M3 P1 P2 P3 P4 

EDUCATION TEACHING 2 1 0 2 1 1 0 

RESEARCH LEARNING 3 2 0 3 2 1 1 

DEVELOPMENT 

ACTING 

REFLECTING 

4 

4 

2 

1 

0 

0 

3 

2 

2 

2 

1 

1 

0 

0 

PRODUCTIONS 4 3 1 3 3 2 1 

COMMUNICATION SERVICES 4 2 1 3 3 3 1 

PARTICIPATIONS 3 3 0 3 3 2 1 

Table 2. Sample of an evaluation of the transformation activities 

As a result of successive applications of the TADIR protocol we now understand much better the outcomes and 

performance of the specific LC of trainers involved in solving the problem: How can we improve a learning 

community of educators working in teacher training? The use of the protocol was also helpful in building a 

functional reservoir for the management of the corresponding organizational knowledge of the human learning 

system in charge of finding the solution: the LC of trainers. However, we must be aware that the TADIR 

protocol is just a useful procedure in order to follow the evolution of a system; it is neither a research 

methodology for generating knowledge nor a rigid recipe for obtaining practical results. The main strengths and 

limitations in using TADIR derive from these characteristics. We have no intentions of evaluating this protocol 

neither to compare with similar procedures used in problem solving. 

Remarks on the use of the TADIR protocol 

In this section we comment on the connections among the TADIR problem solving protocol and two approaches 

of importance in education: action learning which shows the potential of the protocol, and collaborative work 

which reveals a promising direction towards possible improvements in TADIR applications. 

26

• Action learning 

According to McGill & Beaty (2001), action learning means learning through action and reflection. It is an 

effective structured process of learning requiring engagement and reflection in order to make things happen or to 

produce changes. It is also a vehicle for development and change that can occur at the level of the individual, the 

organization or the society. Action learning is a fruitful methodology that serves to focus willingness and 

commitment in order to accomplish human specific tasks in the present, based on reflections that take into 

account both past experiences and future plans for acting. 

Successive applications of the action learning methodology serve to detect and correct situations in which 

different obstacles can hinder or slow down the problem solving process, for instance: conceptual errors, false or 

unnecessary assumptions, improper reasoning, wrong expectations, results obtained under inappropriate 

conditions, simplistic interpretations of results, confrontation with unrealistic goals... Action learning can be 

considered as an intellectual instrument for understanding the process of getting better and better solutions in 

connection with problem solving in human learning systems. It is in this sense that the TADIR problem solving 

protocol is a practical implementation of action learning where the cognitive and metacognitive dimensions of 

the protocol make explicit reference to the dipole of acting to solve a problem and reflecting on the results and 

procedures. 

• Collaborative work 

Teasley & Roschelle (1993) define collaboration as a process in which human beings negotiate and share 

relevant meanings in connection with problem solving tasks. It is a coordinated and synchronic activity resulting 

from building and sharing a common conceptualization of a problem as well as the procedure to be followed in 

order to solve it. These authors assume that collaborative work exists in the collective space defining the 

problem: it corresponds to the definition of a common conceptual structure that supports the activities required to 

solve the problem by integrating goals, descriptions and ideas about possible solution paths. Collaboration is 

different from cooperation, where the problem solving task to be accomplished is reduced to a simple division of 

the work to be done among the members of the group. 

Recently, collaborative approaches in education have been successful connecting the educational requirements of 

the projects under consideration with the advantages of modern technologies. This has been the case with 

Computer Supported Collaborative Learning (CSCL), introduced by Koschmann (1996). In CSCL the central 

issue is human communication and collaboration mediated through the computer, which now plays the role of a 

knowledge-building instrument instead of aiming to replace some capabilities of human tutors. CSCL makes use 

of the constructivist approach and asserts that contextualized knowledge is built through collaborative learning 

processes that require the commitment of participants to join efforts in order to solve problems. 

As CSCL approach has been critical to make the implementation of educators’ plans and decisions more 

efficient, it presents a promising scenario for TADIR applications. When the members of our LC were involved 

in any of the transformation activities corresponding to the conceptual model of the proposed solution, they 

made progress in order to work collaboratively. In pursuing that goal, the TADIR protocol was a useful tool 

serving to find strengths and limitations of the LC in itself, in the sense that the trainers and the trainees needed 

to work more in the direction mentioned by Laffey et al. (2002): “CSCL-type systems can facilitate schools 

becoming learning organizations, not just organizations that support learning.” 

Taking into account that technology improves the way we do things and changes what we do (Norman, 1993), 

two relevant consequences can be considered after working on this application of the TADIR protocol in order to 

develop a technology leadership team: 

(1) the convenience to circumscribe the appropriate boundaries of the protocol applications to human learning 

systems in which effective collaborative conditions are implemented (Johnson & Johnson, 1996), and 

(2) the importance of having a framework structure in order to interpret and follow up successive conceptual 

models for the solution of the problem, like for instance the notion of “Envisionment and Discovery 

Collaboratories”, involving the organization of computerized spaces of action and reflection that support the 

creation of shared understanding through collaborative design (Arias et al., 2000). 

27

Acknowledgement 

The work here reported was partially supported from PAPIIT Grant No. IN 305901 from UNAM. 

References 

Arias, E. G., Eden, H., Fischer, G., Gorman, A., & Scharff, E. (2000). Transcending the individual human mindcreating 

shared understanding through collaborative design. ACM Transactions on Computer Human- 

Interaction, 7 (1), 84-113. 

Baker, M. (2002). Forms of cooperation in dyadic problem solving. In P. Salambier, & H. Benchekroun (Eds.). 

Cooperation and complexity (pp. 1-38). Paris: Hermes. 

Barojas, J., & Pérez y Pérez, R. (2001). Physics and creativity: problem solving and learning contexts. Industry 

and Higher Education. 15 (6), 431-439. 

Barojas, J., Jiménez, E., & Sayavedra, R. (2001). Rethinking distance education. In M. Rahman Sayed, & V. 

Tareski (Eds.), Advances in educational technologies: multimedia, WWW and distance education. (pp. 127-134). 

New York: John Wiley. 

Bates, A. W. (1995). Technology, open learning and distance education. London: Routledge. 

Checkland, P.B. (1981). System thinking, system practice. Chichester. John Wiley. 

Forcier, R.C. (1999). The computer as an educational tool: productivity and problem solving. Upper Saddle 

River. New Jersey: Merrill - Prentice Hall. 

Johnson, D. W., & Johnson, R. T. (1996). Cooperation and the use of technology. In D. H. Jonassen (Ed.), 

Handbook of research for educational communications and technology. (pp. 111-118). New Jersey: Lawrence 

Erlbaum. 

Jonassen, D., Peck, H. K. L., & Wilson, B. G. (1999). Learning with technology: a constructivist perspective. 

Englewood Cliffs, New Jersey: Merril -Prentice Hall. 

Koschmann, T. (1996). Paradigm shifts and instructional technology: an introduction. In T. Koschmann (Ed.), 

CSCL: theory and practice of an emerging paradigm. (pp. 1-23). New Jersey: Lawrence Erlbaum. 

Laffey, J.M., Musser, D.R., Espinosa, L., Remidez, H., Gottdenker, J. S., Hong, R. Y., & Amelun, C. (2002). 

CSCL for schools that learn. In G. Stahl (Ed.), Computer support for collaborative learning: foundations for a 

CSCL community. Proceedings of CSCL 2002. (pp. 1017-1044). New Jersey: Lawrence Erlbaum. 

Linn, M.C., & Hsi, S. (2000). Computers, teachers, peers: science learning partners. Mahwah, New Jersey: 

Lawrence Erlbaum. 

McGill, I., & Beaty. L. (2001). Action learning: a guide for professional management and educational 

development. London: Kogan Page. 

NCREL (2000). North Central Regional Educational Laboratory. Retrived October 2, 2003, from 

http://www.ncrel.org/engauge/skills/indepth.htm#literacy. 

Norman, D. (1993). Things that make us smart: defending human attributes in the age of the machine. Reading, 

Massachusetts: Addison-Wesley. 

Schoenfeld, A. H. (1992). Learning to think mathematically: problem solving, metacognition, and sense making 

in mathematics. In D. A. Grouws (Ed.), Handbook of research on mathematics teaching and learning. (pp. 334 – 

370). New York: Macmillan. 

Teasley, S.D., & Roschelle, J. (1993). Computers as cognitive tools. New Jersey: Lawrence Erlbaum. 

Wiggins, G., & McTighe, J. (1998). Understanding by design. New Jersey: Merril-Prentice Hall. 

Wilson, B. (1990). Systems: concepts, methodologies and applications. Chichester: John Wiley. 

28

Baillie-de Byl, P. (2004). An Online Assistant for Remote, Distributed Critiquing of Electronically Submitted Assessment. 

Educational Technology & Society, 7 (1), 29-41. 

An Online Assistant for Remote, Distributed Critiquing of Electronically 

Submitted Assessment 

Penny Baillie-de Byl 

Department of Mathematics and Computing 

University of Southern Queensland 

Toowoomba, Queensland, 4350, AUSTRALIA 

Tel: +61 7 4631 5521 

Fax: +61 7 4631 5550 

penny.baillie@usq.edu.au 

Abstract 

This paper outlines the architecture for an online assessment management system implemented at the 

University of Southern Queensland. The system assists teams of academics in the management and 

marking of electronically submitted student assignments in large-scale classes. The system designed to 

provide a flexible yet structured method for providing feedback to students also offers semi-automatic file 

handling and grade recording. The system, Classmate, allows a team of markers to access and mark student 

assignments through a web interface designed to parallel paper-based marking systems. An online 

authoring tool replaces the red pen on paper analogy. A pilot study conducted on the use of Classmate has 

found the system to be of use in providing students with consistent feedback, allowing traditional 

assignment interactions and reducing trivial and repetitive assignment marking tasks. 

Keywords 

Online assignment submission management, Authoring tools, Electronic submission, Distance learning, Largescale 

class management 

Introduction 

The University of Southern Queensland is a world leader in distance and online education. The University has a 

student body, in which more than 100 nationalities are represented, with more than 16,000 of its 22,000 students 

of all ages studying in their own locations world-wide. In the department of mathematics and computing, courses 

are taught containing up to 1200 students who either attend classes on campus or obtain teaching materials via 

the Internet. 

As class numbers increase, new teaching methodologies are needed so students do not fall prey to the 

inadequacies of traditional student-teacher based models of interaction (Preston & Shackelford, 1998). On 

campus students in large classes often miss out on the personalised attention given in smaller classroom 

situations and for distance (online) students this type of interaction is logistically impossible. To cope with large 

numbers, a course is usually managed by a team consisting of the examiner (team leader), tutors and markers. 

The traditional approach in handling large class numbers has been to investigate the use of computers in batch 

processing student details. This has seen a natural progression toward Internet-based applications. 

The growth of the Internet has caused dramatic growth in the number of online applications and development 

tools for educators (Lever-Duffy et al., 2003). Developers such as WebCT (http://www.webct.com) and 

Blackboard (http://www.blackboard.com) provide suites of software that assist teachers in developing online 

course content by supplying turn-key applications that offer content presentation, interactive learning 

environments and student activity tracking. The focus of online education has changed from the simple 

reproduction of teaching materials on webpages to the goal of motivating students to become interactive learners 

(Trotter, 1999). 

Contemporary online teaching environments provide a plethora of assessment tools. However, while methods 

such as multiple choice, matching, fill-in-the-blank and formulaic questions lend themselves to online education 

and automatic marking (Goldberg, 2000), the primary disadvantage of these assessment methods is their poor 

ability to accurately reflect student learning (White, 2002). Although it is possible to integrate a more accurate 

means of testing a student’s knowledge and understanding (for example, essays and projects (White, 2002)) into 

an online teaching environment (Ritchie & Hoffman, 1997), the advantages of using contemporary online 

education environments ends after the electronic submission. The expert judgment of the respective examiner is 

essential in marking and critiquing work (Korb et al., 1997). Many online educators shy away from electronic 

submission of essays and projects because of the level of technical expertise required to disseminate, open, 

critique and administer them (White, 2002). 






29

Depending on the course, the format of the submitted file can vary (Word, Excel, MATLAB, text etc.). When the 

submissions are to be marked, the examiner must divide the submissions among markers. This may include 

printing the assignments and distributing paper copies or collating the electronic files and distributing them to 

the markers on disk, via email or a network. Depending on the format of the electronic submissions the markers 

may add comments directly into the submitted file or create a new text document, add feedback and email the 

feedback to the student. The markers record the individual results and when marking is complete, these are 

passed to the examiner. The examiner manually enters these results into a spreadsheet or database system. 

Several shortcomings identified with this method are: 

• Feedback can be inconsistent among markers (Preston & Shackelford, 1998). 

• Markers need to organise and administer email lists. 

• Markers wanting a more traditional approach to marking may print out the submissions, mark them 

with a pen and return via mail. 

• Double handling of student results. 

• The examiner cannot track the progress of marking without elaborate administration and continued 

communication with the markers. 

From studies conducted by Preston and Shackelford (1999), the ideal online marking system would: 

• place emphasis on the student's submitted work 

• allow a holistic view of the submitted work 

• provide quick and easy navigation between sections of the work 

• highlight syntax to improve readability 

• provide markers with a logical and consistent means of critiquing 

• allow for student by student or problem by problem marking 

• separate the interface of the system from the implementation 

• automate grade submission and file downloading and uploading. 

To address these criteria and overcome the current difficulties in managing electronically submitted student 

work, a prototype for an online learning agent system called Classmate, has been developed. For the purposes of 

this discussion an agent is considered a piece of software that works independently of the user to perform tasks 

on the user's behalf (Maes, 1994). Classmate allows a team of markers to assess students’ work in an organised, 

virtual marking and critiquing environment that provides timely, valuable and consistent feedback to the 

students. The system consists of a collection of intelligent agents that semi-automate the marking of 

assignments. 

This paper presents the architecture of the Classmate system and examines the use of the system in the course 

CSC1401 Programming in C. It begins with an introduction to related work in the domain of electronic 

assessment management. This is followed by a detailed description of the system before an example of the use of 

the system is outlined. We conclude by discussing planned future work and system enhancements. 

Related Work 

Developing automated assessment systems is not new in computing disciplines. Examiners who work within the 

computing domain have the necessary skills to design and write programs for their own use. It would be 

uncommon to find a computing department at a university that did not have its own course specific student batch 

processing systems whether they are for automated marking or final grade calculations. These types of narrowfocussed 

systems, while perfect for the course for which they were designed, are difficult to modify for broad 

usage in other disciplines and courses. In addition, as Internet companies have seen the value in developing sets 

of generic teaching tools as learning management systems (LMS), much of the work developed within the 

computing departments has been overlooked but not superseded. 

In 1995, Dawson-Howe (1995) described a command-line system called submit. This system prompted 

computing students to run compilation examples and sample runs (testing) of their programs in the Unix 

environment. As the programs were tested and compiled a script of the session was recorded. When completed, 

the script and the files would be submitted into a central repository for collection by the examiner. This program 

became a standard on most versions of Unix and implemented in many computing departments world-wide. The 

electronic submission component of the submit program has been outdated by the web-based graphical user 

30

interfaces of LMS's such as the dropbox provided by Blackboard. However, the prompting for testing and 

compilation has not. 

Another system that provides students with the means of electronic submission is BOSS (Luck & Joy, 1999). 

This system expands on the ideas from the submit system by automatically testing programs submitted by 

computing science students. The system does not provide automated marking, but does give markers access to 

the program test runs for use in evaluating the student's work. 

Blaine and Petre (1997) describe an electronic marking system that accepts electronically submitted assignments 

and forwards copies of the assignments to a marker. The marker uses Microsoft Word to analyse the work with a 

specially formatted template that presents inserted and deleted parts of the original file in a different font. The 

marker returns the files to the marking system, which returns the feedback to the student, enters the student's 

grade into the university student database and sends the marker a receipt. 

A more structured approach to marking is given in (Preston, 1997) where the Grading Environment (GE) system 

is described. This system designed to improve the reliability and consistency of student performance evaluations 

among a group of markers, guides them through the marking process. The system takes the traditional pen and 

paper based approach to marking and places it on the screen. Each marker is asked a series of evaluation 

questions about the student’s work. Through point and click interaction the system evaluates the student’s work 

and calculates a grade for the student. The system also uses the marker's responses to set remedial work for each 

student based on the errors in their assignments. The system was designed for marking computing students' 

programs although it can be adapted for other courses. Other systems such as the one described by Mason & 

Woit (1999) allow annotations to be entered into the assignment files via a structured webpage. The webpage 

allows the marker to enter a comment into a textfield and this comment is further programmatically inserted into 

the student's submission. This allows single lines of text to be added in the student's file. However, the marker 

cannot correct or modify existing lines. 

Another web-based assignment management system is illustrated by Darbyshire (2002). This system provides 

students with a means of submitting work via the Internet and limits the accepted file formats. It also allows 

markers to organise the files for downloading. This system places more emphasis on the submission and file 

management process than it does on marking. Darbyshire suggests improvements to the system would be 

• the use of annotation tools such as the change tracking facility in Microsoft Word; 

• the ability for markers to resubmit marked files (previously downloaded) back into the system; and, 

• the expansion of content sensitive interaction by allowing more file formats to be submitted. 

Classmate builds on the ideas developed in the before mentioned systems, marrying electronic submission and a 

grading environment with file management, course team management and automated student feedback in an 

online environment. It includes an online annotation tool, handles all file formats and allows markers to upload 

and download files. It provides both a free format method of critiquing students' work along with a structured 

grading template (similar to Joy & Luck, 1998). Unlike systems such as Ceilidh (Benford et al., 1994) and the 

semi-automated system described in Jackson (2000) which endeavour to provide facilities to both the student and 

the marker (including automated marking), Classmate focuses on the marker's needs and has therefore been 

developed with facilities not currently available in the before mentioned systems. Classmate has been developed 

with free format assignments in mind and does not require rigid formatting rules to be adhered to by the student 

as are needed in current automated marking systems. The architecture of this system will now be discussed. 

Overview of Classmate 

Classmate currently resides on the web server in the Department of Mathematics and Computing at USQ. The 

system was written using a mixture of C/C++, Perl and Java to give it the greatest flexibility. The system 

consists of 4 main agent entities. The main server-side agent works to manage student submissions, administer 

student receipts, distribute submission to markers and collate marker feedback. The smaller agents are 

distributed as necessary and act as clients on the user's machines. The agents are independent of any database 

system and currently reside over a file/directory structure. This makes the system modular (other system parts 

can be added) and platform independent. 

A conceptualisation of the Classmate system is given in Figure 1. 

31

Figure 1. The Classmate System 

In short, the steps involved in using the system are as follows: 

1. The student submits an assignment electronically via the Internet. 

2. The examiner sets the marking criteria. 

3. The examiner assigns a marker to the assignment. 

4. The marker marks the assignment using the examiner’s marking criteria. 

5. The marker inserts free formatted comments and suggestions into the student’s assignment file. 

6. The student’s mark, a listing of the criteria and the edited assignment file are emailed back to the 

student. 

7. The student’s mark is uploaded to the university’s student database. 

These steps will now be discussed in detail. 

The use of Classmate begins with Agent A being used by the examiner to setup the course details. This agent 

asks the examiner a number of pertinent questions, such as the number of assignments, the assignment due dates 

and the names and ids of the markers. The examiner must also provide the agent with a file containing a list of 

the students enrolled in the course and their contact details (including an email address). For each assignment the 

examiner can give Agent A a list of the marking criteria for an assignment along with the associated sub-marks 

for each. The examiner can also provide a solution file with answers to the assignment. The view presented to 

the examiner by Agent A is called the Briefcase (shown in Figure 2). 

In the Briefcase view, the examiner can see a list of their students and the status of each assignment. If an 

assignment is waiting to be marked, it is displayed as a folder icon. If the assignment has been marked, a result 

will appear in the appropriate column. The Classmate system also allows the examiner to set cutoffs for grading 

and these are calculated automatically and displayed. 

32

Figure 2. Briefcase View of Classmate 

When a student wishes to submit an assignment, they download Agent B. This agent uses their student login id 

to access the universities central student database using LDAP (Carter, 2003). The agent displays these details to 

the student and provides them with a list of assignments and due dates from which to choose and submit their 

work. The student's details and submitted files are streamed to the Classmate server-side agent for storing in a 

repository located, for security reasons, outside the realm of the web server. Because of Classmate's modular 

nature, Agent B is replaceable with other file submitting software. For example, one examiner has written his 

own script for his students to submit files. For future versions of Classmate, the WebCT and Blackboard 

dropboxes are being examined as replacements for Agent B. 

Before a marker can access the students' submissions and commence marking, the examiner must interface with 

Agent A in order to allocate students to specific markers. Until this is done, Agent C will report to the marker 

that they have no assignments to mark. Agent A will allow the examiner to manually assign students to markers, 

however for very large classes where this is impractical, the examiner can simply tell the agent what the 

maximum allocation (in numbers) for each marker is and the agent will randomly assign students to markers. 

Having done this, the agent reports back to the examiner the actual number of students that were allocated to 

each marker. At this time the examiner has the opportunity to make manual adjustments to the allocations as they 

see fit. 

Once students have been allocated to a marker, the marker can obtain a view of their student list from Agent C. 

The view, called the Marker's Briefcase, is similar to the examiner's Briefcase of Figure 2 but with restricted 

administration functionality. The assignment critiquing process is twofold. Firstly, the marker can make 

corrections and feedback directly into the submitted file using free formatted text and secondly a grade can be 

calculated by following an examiner's marking criteria. 

The marker can use the mouse to select a student's work to mark (by clicking on the folder icon). On selection, 

Agent C presents the marker with a list of submitted files (shown in Figure 3). If the submitted file is a zip file, 

the marker can ask the agent to unzip the contents of the file. When this request occurs, Agent C contacts the 

server-side agent who performs this task and provides Agent C with an updated list of the unzipped files to 

33

present to the marker. The marker can view the contents of the file, through Agent C, by clicking on the 

filename. Agent C then requests the contents of the file from the server-side agent. The server-side agent 

accesses the file from the repository and streams it to Agent C for display. The file is displayed to the marker in 

an authoring textbox (shown on the right in Figure 3). For security reasons, the marker cannot modify the 

originally submitted file and therefore can ask Agent C to request a copy of the file for editing. Agent C relays 

this request to the server-side agent. The server-side agent creates a copy of the original file and transforms the 

file into Rich Text Format (RTF). The RTF file is sent back to the marker via Agent C. The RTF file was chosen 

as it allows different coloured text to be entered. As the marker critiques the student's file with comments and 

corrections, any text entered by the marker appears in red. Currently Agent C is capable of presenting plain text, 

HTML and RTF in its authoring textbox. For other submitted files, the marker has the option of downloading 

them to their own computer and opening them with the appropriate software (e.g. Word or Excel Documents). If 

the marker makes corrections to these downloaded files on their own computer, they can upload them via Agent 

C back into the repository. 

The major difference between Classmate and other dropbox type electronic submission systems is that the 

submitted work stays in the repository during the marking process. This means that the marker knows where the 

file is at all times (not in a sub directory on their own computer or on a floppy disk in their office). Unlike in a 

paper based system if the marker goes missing (Jones & Behrens, 2003) the assignments do not go with them. 

Feedback and error corrections are entered directly into a copy of the submission. This copy also resides in the 

repository. The result of managing the marking of student submissions in this manner means that the marker's 

comments and notes are recorded with the submission and associated grade. This differs from traditional 

marking methods where annotations written on paper that describe and justify a student's grade for an 

assignment are usually lost (Preston, 1997). 

Figure 3. Marker's view of submitted files and freeform comment authoring 

34

In addition to appraising the files in the authoring textbox, the marker also completes a marking criteria sheet 

(shown in Figure 4). This sheet is generated by Agent C from the criteria entered in by the examiner. The marker 

fills out the sheet by clicking on tick boxes and entering numeric values. Once completed, Agent C calculates a 

total number of marks for the student and sends the details to the server-side client for storage. The marker or 

examiner can revisit the marking of an assignment at anytime and make adjustments as necessary. The total 

calculation also takes into consideration any extensions given by the examiner. Agent C obtains extension 

information and the time stamp of the student files, compares them and calculates a deduction if applicable. 

Figure 4. Marking Sheet 

When the marker is satisfied with the assignment mark, they can inform Agent C to complete the electronic 

assessment cycle by sending the assignment feedback to the student. This process begins when the marker clicks 

on an envelope icon displayed next to the students work (see Figure 2) in the Marker's Briefcase. Agent C sends 

this request to the server-side agent. The server-side agent accesses the student's files and critiquing, generates a 

feedback letter and emails this letter along with any marker corrected files and the sample solution to the student. 

Discussion 

The Classmate system has been in use since 1994 in differing formats. The latest evolution (as discussed in this 

paper) was implemented for the first time in semester 1 of 2003. To date, the Classmate system has been used 

by ten markers in four different courses. Of these, six were available to be surveyed on the characteristics of the 

system using Preston and Shackelford's (1999) ideal online marking environment qualities (as listed in the 

Introduction) as benchmarks. In addition to these qualities, the markers were asked to evaluate Classmate's 

35

quality and quantity of student feedback. It should be noted that this survey constitutes a pilot study on the 

acceptance of the system. The small population provides insufficient power to discriminate adequately on 

statistical significance. The results are nonetheless helpful in guiding future developments of Classmate. 

For each characteristic the markers had to indicate their agreement or disagreement by answering with Yes or 

No. After each question, the markers were given the opportunity to explain their answer. The questions and 

results are shown in Figure 5. 

For the majority of the criteria the markers agreed that Classmate possessed the characteristics of an ideal online 

marking system. 66% of the markers found the Classmate system reduced some of the repetitive and trivial tasks 

associated with marking. One of the markers who answered No to this question commented: 

“The same marking still needed to be done. This system allowed the trivial and repetitive tasks to be done more 

easily.” 

50% of the markers commented that the annotation system allowed them to cut and paste frequently used 

comments, thus reducing marking time. They performed this task using an opened text editor and cut and paste 

comments from the editor into the submission. In the newer version of Classmate, Agent C keeps track of the 

marker's annotations and provides the marker with a drag and drop list with the authoring textbox shown in 

Figure 3. 

Does the Classmate system: 

Marker's Evaluation of Classmate Characteristics 

reduce the repetitive and trivial tasks associated w ith marking? 

allow you to mark assignments anyw here and anytime? 

reduce administrative task associated w ith managing assignment distribution, 

marking and responses? 

allow your marks to be consistent, meaningful and accurate among students? 4 

reduce erroneous data entry errors associated w ith entering grades into 

university databases? 

increase the amount of feedback you give to students? 

increase the consistency of feedback you give to students? 

increase the quality of feedback you give to students? 

place emphasis on the students submission? 

allow for a 'big-picture' view of the student's w ork, allow ing you to hide or 

display implementation details w hen needed? 

allow for a quick and easy navigation betw een sections of the students 

work? 

enable easy annotation features for providing feedback to the student? 

automate mark/grade submission? 

allow for uploading and dow nloading of files? 

allow you to mark the submitted w ork w hile keeping w ithin the examiner's 

marking criteria? 

reduce your marking time of individual assignments as compared w ith 

other methods you have experienced? 

Figure 5. Classmate Evaluation 

1 

2 

3 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

Y 

N 

0.00% 50.00% 100.00% 

36

66% of the markers agreed the Classmate system allowed them the freedom to mark assignments anywhere 

anytime. Of the markers that disagreed, one stated: 

“Previously, the system was too cumbersome to be used via a dial-up connection. The new system is faster and 

should allow this.” 

It has been noted that while the Classmate system works successfully over the University of Southern 

Queensland's high speed internet connection, it has been found much slower over a dialup connection. While 

efforts have been made to reduce the size of the Classmate webpages for faster viewing the size of student 

assignments cannot be controlled as easily. 

Five of the markers agreed that Classmate provided a consistent, meaningful and accurate way of assessing a 

student's assignment. One marker disagreed and commented that the marking criteria provided was not detailed 

enough. It should be noted that the supplied marking criteria are setup within Classmate by individual examiners 

and the lack of detail specified in a marking criteria webpage (shown in Figure 4) is the responsibility of the 

examiner and not a shortcoming of the Classmate system. 

Five of the markers agreed that Classmate gave them a 'big-picture' view of the student's submitted work and 

allowed them to navigate between sections of the assignment. For the one marker who disagreed, the reason 

given was that they had not used these features of Classmate. As it is unclear how a marker using Classmate 

could not have used these features, it is assumed that they did not understand these questions. 

All markers agreed Classmate performed favourably in reducing administrative tasks, placing emphasis on the 

student's assignment, providing annotation features for feedback, automating grade submission to the universities 

databases and marking within the examiner's criteria. In addition, all markers agreed that the Classmate system 

reduced their marking time. The markers reported a reduction in assignment turnaround time from 3 weeks for a 

paper based system to 4 days using Classmate. This is not unusual in other electronic submission management 

systems (Price & Petre, 1997) and it could be fanciful to think that a further reduction in this timeframe is 

possible. In this investigation some markers reported they did not start marking the assignment as soon as it 

arrived. This meant that the turnaround time was increased simply because the submitted file sat idle on the 

server for a couple of days (or even weeks in some cases). One keen marker had a turnaround time of 1 day. This 

was made evident in a class after the submission due date when a couple of students commented with surprise 

that they already had their assignment marks back. 

Finally, all markers agreed that Classmate reduced the amount of erroneous student data as results are recorded 

within the system at marking time and uploaded to the university systems. The current version of Classmate 

uploads data into the universities Peoplesoft database whereas examiners using the WebCT platform still need to 

manually transfer their marks out of WebCT and into Peoplesoft. 

In addition to these evaluations, the markers were asked to rate Classmate against other assignment handling 

methods they had experienced. The methods compared were pen and paper (handling of assignments submitted 

on paper), manual submission (assignments submitted on CDROM or floppy disk) and email (assignments 

emailed directly to the marker or examiner). They were asked to rate Classmate on a 5 point scale (‘1’ begin 

Significantly Poorer to ‘5’ being Significantly Better) in the areas of assignment turn-around time, feedback 

quality, feedback consistency and feedback quantity. The results are displayed in Figure 6. 

As can be seen in the graph of Figure 6, in all areas for all handling methods Classmate performed favourably. 

Classmate was rated better and significantly better in all categories when compared to the handling of manually 

submitted electronic files and emailed files. Classmate was also rated favourably against paper based 

submission. As paper based submissions allows markers much greater flexibility in marking it was gratifying to 

have the system accepted as a more acceptable means of assignment handling. 

For the purpose of evaluating the Classmate system from the students' point of view, 27 students were surveyed 

on the turnaround time, quality and quantity of the feedback they received. They were asked to compare 

Classmate with the same assessment handling methods evaluated by the markers using the same scale. The 

results were inconclusive as the students' experience with other assignment handling methods was limited. Only 

20 of the students could compare Classmate with paper-based submission, eight students could compare with 

manual electronic submission and seven students could compare with emailed assignments. On average students 

rated their feedback from the system as the same or better than other systems. 

37

However, as the students could only evaluate the system based on what they experienced of it (the feedback) it 

was difficult to gauge if the students were evaluating Classmate or the feedback given by their marker. Although 

the examiner can specify the marking criteria used by the marker, they cannot control the amount of free 

formatted guidance given to the students by individual markers. It was suspected that students were evaluating 

the system based on which marker assessed their work when comments about the system included personal 

criticism directed at their marker rather than the actual system. Comments made included: 

“Increased amounts of feedback are beneficial to future assignments. In my case I only received 2 lines of 

comments for the 10 marks that I lost in my code. The comments were very brief. I think it would be more 

beneficial to hand write comments and feedback so you can pinpoint exactly what is wrong with your code”. 

and 

“Your feedback [NAME OF MARKER] and responses for the single assignment were excellent and 

satisfactory.” 

These two extremes in student opinion suggest that some markers may not have explored the full potential of the 

Classmate system and used all the facilities available. This implies that further training for the markers in the 

system’s use is needed. 

5.00 

4.50 

Mean Evaluation 

4.00 

1 - Significantly Poorer 

2 - Poorer 

3.50 

3 - Same 

4 - Better 

3.00 

5 - Significantly Better 

2.50 

Pen & Paper 

Manual Submission 

Emailed 

2.00 

1.50 

1.00 

0.50 

0.00 

Mean Comparisons of Classmate 

with other Assessment Handling Methods 

Turnaround 

Time 

Feedback 

Quality 

Feedback 

Consistency 

Feedback 

Quantity 

Figure 6. Marker's Comparison of Classmate with other Assignment Handling Methods 

Further to comparing Classmate with other systems, the students were asked for their opinion of the turnaround 

time for marking and the method used for assignment feedback. The answers were given on a five point scale 

from very disappointing to very acceptable. The results are displayed in Figure 7. 

These preliminary results suggest the majority of students are happy with the use of Classmate in assessing their 

assignments. More evidence the students were tending to evaluate their experience with their marker and 

examiner rather than the Classmate system can be seen in Figure 7. The student who rated the method of 

feedback as very disappointing gave the reason: 

“Unfortunately there has been NO virtual feedback on any of the tutorials or work leading up to this - resulting 

in a diminished result for my efforts on the later marked assignment. If the system was more responsive for ALL 

work and not just some I would be both very impressed and very much helped by it.” 

38

In the above comment, the student referred to parts of her course that were not integrated with Classmate. It is 

evident that she was impressed with the Classmate system used for marking her assignment and she wished it 

could have been used for other work involved in the course. 

Disappointing 

Barely 0% 

Acceptable 

8% 

Acceptable 

46% 

Very 


0% 

Very 

Acceptable 

46% 

How would you rate the feedback 

timing? 


Barely 8% 

Acceptable 

8% 

Acceptable 

35% 

Figure 7. Student's opinion of Classmate system 

Very 


4% 

Very 

Acceptable 

45% 

How would you rate the method of 

feedback used for this course? 

As the Classmate system is mostly transparent to the students it is difficult to gauge their acceptance of the 

system. What they expect in the feedback to their assignments still lies in the hands of the examiner and 

markers. With Classmate's flexibility, the examiner can create as elaborate marking criteria sheet as they wish 

and with the online annotation system the marker can add as little or as much feedback as they see fit. 

The success with acceptance of Classmate among markers can be attributed to the interface design being 

modelled on the paper based system. Classmate attempts to add value to the electronic submission management 

and marking process with semi-automated processes while allowing markers to interact with the students' work 

in a traditional manner. 

The Classmate system has proved useful in: 

• reducing repetitive and trivial assessment related tasks (such as computer code compilation); 

• providing markers with the flexibility to mark assignments anywhere, anytime without having to have 

all the submitted files on their own computer; 

• managing the administration of electronic submission among a group of markers; 

• providing an interface that allows markers to mark an assignment via a coherent interface (no matter 

what the file format) in a traditional manner (red pen on paper) and keeping within the examiners 

guidelines; 

• presenting real time information on the progress of assignment marking and the status of student 

submissions to the examiner via the web interface; and 

• reduces erroneous student grade data through the agents’ management of results collection, the emailing 

of feedback to students and the uploading of result data directly into appropriate university 

administration systems 

Conclusions and Future Work 

Classmate has proven affective in providing examiners and markers with a semi-automated electronic 

submission management system. Classmate endeavours to streamline the process of marking electronically 

submitted assignments while not placing restrictions on the content of these assignments. To date, Classmate 

has been used primarily for the marking of programming code as it was developed in the environment of a 

computer science department. The latest version of Classmate is now being trialed by a surveying examiner for 

the management of Excel spreadsheet assignments. 

39

Classmate's independence from a database and platform would make it ideal for integration with WebCT and 

Blackboard like systems. For example, the student list displayed in the briefcase could be easily populated with 

an API call to the Blackboard system. The latest version of WebCT (Vista) allows network drive mapping to 

areas of the system that store the student’s uploaded files. As the location and structure of these files is known 

they could easily be streamed by the server-side agent to Agent C for marking management. 

The focus for future developments of Classmate will be to upgrade the agents. While no specific agent 

architecture has been identified for development, it is planned to implement a learning mechanism in the 

marking Agent C. Over time, the agent will learn to identify repetitive mistakes in assignments and be able to 

add commentary feedback to assignment files before the marker attends to them. For example, a common 

mistake in CSC1401, Programming in C, is the improper use of recursive programming. The agent, with 

assistance from the marker, will learn how to identify recursion and subsequently semi-automate the marking of 

a group of assignments by pre-assessing and adding appropriate feedback. After an agent learns a critiquing 

procedure from one marker, it will communicate this with other marker’s agents via a central knowledge base, 

thus distributing the knowledge of one marker among the marking team. The distribution of expert knowledge 

within the agents and the storage of this knowledge would then be available for use in marking future assessment 

tasks and new offerings of a course. 

Another area identified for inclusion in the system would be offline processing. Currently the Classmate agents 

work by having continued access to the server. In some situations this is not ideal and markers need to be able to 

mark assignments while offline. Improvements to Agent C will allow the agent to download copies of students’ 

work and the examiner’s marking criteria to the marker’s local machine. The same marking process would be 

adhered to however feedback and marks would be reported locally. When Agent C senses it is online it would 

be able to synchronise the locally marked work with that on the server. 

Although Classmate and systems like it have been in use for the past ten years, not a lot of progress has been 

made in moving the domain of online assessment management systems forward. It seems that duplicate and 

hybrid systems evolve from time to time, but without any real progress. The future task of researchers in this 

domain should be to examine the scope of use for such systems beyond the confines of computer science 

departments and first year programming subjects and to scrutinise the requirements of examiners from other 

subject areas on their current processes and needs. Only then will such systems and methodologies find their way 

into commercially available learning management systems. 

Glossary 

As it is common for different Universities to refer to the same entities using differing terminology, for clarity the 

contents of this paper is accompanied with a definition of the language used herein. A single class in which a 

student is enrolled is referred to as a course, the manager of this course is called an examiner and a marker is a 

person that evaluates a student's work. The term marking is defined as a process undertaken by a marker in 

which a student's work is critiqued for the purpose of determining a grade. Critiquing is taken to mean the 

process by which a marker uses a specific set of criteria, defined by the examiner, to assess a student's work and 

provide relevant feedback and grading. 

Acknowledgements 

Acknowledgements are made to Professor Malcolm McKay and DVC Professor Jim Taylor for the monetary 

support put towards the development of Classmate. Thanks must also go to Ron House, Richard Watson and 

Leigh Brookshaw for their ideas and support of this project and Shirley Reushle, Ivan Wolski and Michael de 

Raadt for their continued use and testing of the system. Further acknowledgement goes to the reviewers of this 

paper for their constructive suggestions for preparing the final manuscript. 

References 

Benford, S. D., Burke, E. K., Foxley, E., Gutteridge, N., & Zin, A. M. (1994). Ceilidh as a Course Management 

Support System. Journal of Educational Technology Systems, 22, 235-250. 

Carter, G. (2003). LDAP System Administration, Sebastopol. O’Reilly & Associates, Inc. 

40

Darbyshire, P. (2000). Distributed web-based assignment management. In A. Aggarwal (Ed.), Web-based 

learning and teaching technologies: Opportunities and challenges. Hershey: Idea Group Publishing, 198-215. 

Dawson-Howe, K. (1995). Automatic submission and administration of programming assignments. ACM 

SIGCSE Bulletin, 27 (4), 51-53. 

Goldberg, M. (2000). Online Testing: Some Observations. Retrieved October 9, 2003 from 

http://www.webct.com/OTL/ViewContent?contentID=2339239. 

Jackson, D. (2000). A Semi-Automated Approach to Online Assessment. In Proceedings of the 5th Annual 

Conference on Innovation and Technology in Computer Science Education. Helsinki, Finland, 164–167. 

Jones, D. & Behrens, S. (2003). Online Assignment Management: An Evolutionary Tale. In Proceedings of 

HICSS03. Hawaii. Retrieved October 9, 2003 from http://cq-pan.cqu.edu.au/david-jones/Publications/ 

Papers_and_Books/OLA_Evolutionary/. 

Jones, D. & Jamieson, B. (1997). Three Generations of Online Assignment Management. In R. Kevill, R. Oliver, 

& R. Phillips (Ed.), What Works and Why, In Proceedings of ASCILITE 97, Perth, Australia, 331-345. 

Joy, M. & Luck, M. (1998). Effective Electronic Marking for On-line Assessment. In Proceedings of ITCSE98. 

New York/Dublin: ACM Press, 134-138. 

Korb, K., Kopp, C., & Allison, L. (1997). A Statement on Higher Education Policy in Australia. Technical 

Report CS97-318. Retrieved October 9, 2003 from http://www.csse.monash.edu.au/ publications/1997/tr-cs97- 

318/policy.html. 

Lever-Duffy, J., McDonald, J. & Mizell, A. (2003). 21st The Century Classroom: Teaching and Learning with 

Technology. Boston: Allyn and Bacon. 

Luck, M. & Joy, M. (1999). Computer-Based Submission and Assessment in BOSS. Interactions, 2 (3), 

retrieved October 9, 2003 from http://www.warwick.ac.uk/ETS/interactions/vol2no3/joy-luck.htm. 

Maes, P. (1994). Agents that reduce work and information overload. Communications of the ACM, 37(7), 30-40. 

Mason, D. & Woit D. (1999). Providing Mark-up and Feedback to Students with Online Marking. ACM SIGCSE 

Bulletin, 31(1), 3-6. 

Preston, J. & Shackelford, R. (1998). A System for Improving Distance and Large-Scale Classes. ACM SIGCSE 

Bulletin Inroads, 30 (4), 193-198. 

Preston, J. & Shackelford, R. (1999). Improving On-line Assessment: an Investigation of Existing Marking 

Methodologies. In Proceedings of ITCSE99. Cracow, Poland, 29-32. 

Preston, J. (1997). Evaluation software: improving consistency and reliability of performance rating. 

Proceedings of ITiCSE '97. Uppsala, Sweden. New York/Dublin: ACM Press, 132-134. 

Price, B. & Petre, M. (1997). Teaching Programming through Paperless Assignments: an empirical evaluation of 

instructor feedback. ACM SIGCSE Bulletin, 29(3), 94-99. 

Ritchie, D., & Hoffman, D. (1997). Incorporating Instructional Design Principles with the World Wide Web. In 

Khan B. (Ed.), Web-based Instruction. Englewood Cliffs, New Jersey: Educational Technology Publications, 

135-138. 

Trotter, A. (1999). From Science and the Workplace Come New Tools of the Trade. Technology Counts ’99, 

retrieved October 9, 2003 from http://www.edweek.org/ sreports/tc99/articles/tools.htm. 

White, J. (2000). Online testing: The Dog Sat on My Keyboard, In Proceedings of the 13th Annual ICTCM, 

Atlanta, USA, London: Addison-Wesley, retrieved October 9, 2003 from http://archives.math.utk.edu/ICTCM/ 

EP-13/C3/html/paper.html. 

41

Pantano Rokou, F., Rokou, E., & Rokos, Y. (2004). Modeling Web-based Educational Systems: process Design Teaching 

Model. Educational Technology & Society, 7 (1), 42-50. 

Modeling Web-based Educational Systems: Process Design Teaching Model 

Franca Pantano Rokou 

University of Aegean, Department of Cultural Technology &Communication 

Distance Learning Laboratory, Samphous 5 

81100 Mytilene, Greece 

Tel: +30-2251-036614 

Fax: +30-2251-036609 

frokou@ct.aegean.gr 

Elena Rokou 

University of Aegean, Central library 

Software Development Department, Imvrou 3 


Tel: +30-2251-036072 

erokou@aegean.gr 

Yannis Rokos 

University of Aegean, Department of Cultural Technology &Communication 

Distance Learning Laboratory, Samphous 5 


Tel: +30-2251-036614 

Fax: +30-2251-036609 

g_rokos@ct.aegean.gr 

Abstract 

Using modeling languages is essential to the construction of educational systems based on software 

engineering principles and methods. Furthermore, the instructional design is undoubtedly the cornerstone 

of the design and development of educational systems. Although several methodologies and languages have 

been proposed for the specification of isolated educational multimedia systems, none has optimum results 

for the description of these systems and, especially, for their pedagogical aspect. Of course this is due 

primarily to how these systems function and are applied; it is not due to the language itself, although its 

special characteristics contribute substantially to the development of these systems sometimes positively 

and sometimes negatively. In this paper, we briefly describe the introduction of stereotypes to the 

pedagogical design of educational systems and appropriate modifications of the existing package diagrams 

of UML (Unified Modeling Language). The main objective of these new stereotypes is to describe 

sufficiently the mechanisms of generation, monitoring and re-adapting of teaching and student’s models 

which can be used in the educational applications. 

Keywords 

eLearning, Web-based Educational systems, UML, educational model, teaching model, instructional design 

Introduction 

Education is undoubtedly the cornerstone of every society. Thus, in every developed society the achievement of 

optimal education and the provision of all needed means to every individual to be educated is fundamental. For 

many years now, major emphasis has been given on the development of educational systems that would be based 

on or supported by intelligent technologies and the World Wide Web. Usually, we distinguish two basic 

categories of educational applications: those that complement and support the tasks of the traditional teacherbased 

classroom, and those that function independently, providing self-learning environment. In both cases, the 

role of the Internet is very important because it provides easy flow of communication and contains a large 

amount of information. The target in most cases is to release and free educational process from time and place 

constraints and to optimize the performance of participants through an individualized education. 

In every web-based application we distinguish three basic levels: the web character of the program, the 

pedagogical background and the personalized management of the learning material. 

The term ‘web-based program’ carries several different meanings in the scientific community. Some scientists 

believe that it refers to those applications that make use of Java, others think that it is anything that uses a web 

server, while a view that is between the aforementioned seems to have prevailed. 






42

In this article, we define ‘web-based program’ as information system that contains a web server, a network, an 

HTTP and a browser in which data supplied by users act on the system’s status and cause changes (Conallen, 

1999). When we refer to the pedagogical background, we mean the educational model that will be used in 

combination with pedagogical goals that are set by the educator every time. More specifically, we refer to 

methodologies that will be used to making the trainees acquire knowledge and will lead them to develop 

particular skills (Lionarakis & Rokou, 2000). For that purpose this educational model uses patterns and 

processes of the physiology of human mind, ways of conceptualization and comprehension of pedagogical 

strategies as well as learning and teaching theories (Rokou, 2002). When we refer to personalized management 

of the learning material, we mean the set of rules and mechanisms that we use to select the learning material 

based on the student’s characteristics, the educational objectives, the teaching model and the available media. 

Framework 

From the above, it is conspicuous that the development of web-based educational systems is much more different 

from the developing process of software in a variety of ways. Firstly, there are many people from different 

backgrounds and skills -such as educators, authors, layout designers, programmers and multimedia experts- 

involved in this process. Secondly, the role of the user becomes stronger and, eventually, makes difficult to 

determine the functional and non-functional requirements of the system. Thirdly, the non-linear structure of this 

kind of applications combined with importance of interactivity, increases the complexity of the design process 

and, of course, the fault rate. Finally, this sort of applications must take into account factors like aesthetics that 

have no relevance to any traditional kinds of software (Nanard & Nanard, 1995). 

At this point, we should note that by saying “traditional software” we mean those programs which are computer 

applications to office management rather than to special domains an disciplines such as Medicine or Education. 

It is well known that in the case of “traditional software”, the development process of software is quite common 

in all applications which are produced and must be more analytical, starkly scalable and recurrent than that of 

traditional systems. Moreover, the maintenance and the frequent renewal of the material of this software are 

basic requirements, whereas in traditional systems, they do not need to be so frequent and their lack does not 

make the software useless. Furthermore, all those people who participate in the production team and the 

structure of the program, based on links, nodes and continual readjustments make demands on modeling 

procedures and stereotypes used (Hennicker & Koch, 2002). 

Furthermore, there is an emerging need to combine and integrate different kinds of learning systems that has led 

to several standardization projects. These projects focus on determining standard architectures and formats for 

learning environments, such as Learning Technology Systems Architecture or Instructional Management 

Systems Project (EDUCAUSE, 1999). Their main issue is the standardization of learning and teaching methods 

(Adelsberger et al., 2000). 

IMS defines and delivers interoperable, XML-based specifications to exchange learning content and information 

about learners among learning systems. Their aim is the modelling of how the systems manage, transfer and 

distribute data relevant to the educational process. 

For our project, we could have used a sophisticated educational modeling language from those available, like 

OUNL-EML or its descendant IMS (IMS Learning Design, 2003). Of course, the primary objective of IMS 

project is to provide: 

• standards in order to locate and operate and interactive platform-independent materials; 

• support for a collaborative and dynamic nature of learning; and 

• incentives and a structure to develop and share content. 

More specifically, in order to attain this goal, the teams of IMS have been developing specifications for the 

following areas: 

• Metadata, the labeling of educational materials; 

• Content, the actions and responses that IMS-compliant content may perform; 

• Management functions, such as access control, session management, tracking students’ progress 

through learning processes, control over the virtual learning environment, and security (Naeve A., 

1999). 

43

Thus, definitions like learning sequences and their attributes (which characterize the content) learning objectives 

and learning materials are being standardized in the Learning Object Metadata Model by the Learning 

Technology Standards Committee of IEEE (LTSC) and the Instructional Management Systems Project 

(EDUCAUSE, 1999). The goal of this standardization is to develop a format for interchange, reuse, and 

combination of learning contents. However, a very important criterion to choose an alternative from a variety of 

learning sequences is the teaching method which is used for specific learning contents. The above-mentioned 

standard approaches do not cover a detailed description of teaching methods. Usually, only information is added, 

which does not cover all aspects of a method and its usability for certain contents (Adelsberger et al., 2000). 

Furthermore, if these standards are used, there will be no common ground of understanding between computer 

scientists and educators. This occurs because the main emphasis is aid on pedagogical and educational aspects 

without providing clear and easy to use correspondence between the pedagogical, educational and technical 

aspects of the under construction system. 

Furthermore, the specific modeling languages require background knowledge of both Education and Computer 

Science if they are to be used effectively. So, due to the fact that it is very difficult to find people quite familiar 

with both fields at the same time, both the modeling and the design of educational software requires the use of a 

modeling language which can be a common ground for and understood by all people involved. 

From the above, it is obvious that the very nature of the educational software and the complexity of the exhisting 

educational modeling languages makes the use of object-oriented (ΟΟ) methods an essential prerequisite for the 

planning and modeling of developing procedures, and this the UML (Unified Modeling Language) seems to be 

the most appropriate for the notational description (Berner et al., 1999). Undoubtedly, up to now most efforts to 

use UML in such specialized categories of software have shown that the specialization of the offered models and 

procedures for the best performance and the fastest designing is necessary (France et al., 1997). 

In this article, we focus on the introduction of stereotypes to the pedagogical design of educational systems and 

of appropriate modifications of the existing package diagrams of UML (Booch et al., 1999), so that these 

stereotypes describe sufficiently the mechanisms of generation, monitoring and re-adapting the teaching and 

student’s models that pedagogical applications have been using. However, we are still working on modeling 

patterns the whole process of educational software production and the expansion of stereotypes, which are based 

on these patterns. 

The presentation in this paper is structured in the following way: We started with an Introduction and a 

Framework. In Section 3, we make an overview of the educational model and its components (student’s model, 

educational goals and pedagogical strategies). In Section 4, we define and describe semantically those 

components required for the design of collaboration diagrams, while providing extension for more use. In 

Section 5, we use the newly added stereotypes to describe the development process of a teaching model. In 

Section 6, we show how the application of the new stereotypes can be used in the production of a simplified 

teaching model. Finally, we summarize what we have achieved and outline future prospectives. 

The educational model-Expansion of stereotypes 

The educational model is the description of theories, principles and processes that aim at standardizing teaching 

processes and regrouping methods. The process of definition of an educational model (Rokou, 2002) is 

illustrated in Figure 1. 

As we see, the process of defining an educational model includes 3 stages: 

1. The definition of the student’s features; 

2. The definition of educational goals; and 

3. The definition of pedagogical strategies. 

The definition of student’s features aims at the configuration of a model for each student, that is, for each student 

data –such as his/her learning styles, his/her knowledge and skills -are collected (Rokou, 2002). The reason why 

this process must take place is relatively simple. We assume that the selection of a suitable educational model 

requires the particular characteristics of each student, so that individualized learning is implemented. In fact, 

extracting the student’s model simulates the process made by a traditional teacher, who asks his/her student and 

makes assumptions based on the way that the latter responds to the former’s stimuli during the first few lessons. 

44

Then, the definitions of educational goals take place simultaneously. They depend mainly on the kind and the 

subject of learning material and on the abilities and the skills we want students to acquire and enhance (Pantano 

Rokou, 2002). 

The educational goals are: 

Figure 1. The processes of definition of an educational model 

• specific, that is the teaching subject (i.e. the scientific domain), the thematic unit and the specificities of 

the teaching material; and 

• general, that is the type of skills that we want the students to acquire; for example, problem solving, etc. 

Finally, after having determined the student’s model and delineated the educational goals, we select the most 

appropriate pedagogical strategy (Pantano Rokou, 2002). In order to achieve the most efficient educational 

model, we select teaching theories compatible with each student’s model. Based on this educational model, the 

course is implemented. 

To sum up, we take the process that we have followed for the production of an educational model based on the 

production of training units. Our objective in this section has been to describe this process in a nutshell in order 

to make clear the particular character of this model and the need of specialized modeling using suitable 

stereotypes in this particular case. 

Notation 

The first and most basic action is the creation of appropriate symbols for the description of the used pedagogical 

and learning models. In this article, we will concentrate on the definition and the semantic description of the 

components required for the design of the collaboration diagrams, while we provide extension for more use. 

45

Initially, we have located the primitive components which take part in every teaching model and their basic 

characteristics. Depending on the teaching model of Figure 1 and on all those referred to Section 3, we can easily 

deduce that the main components are: 

• Student’s model; 

• Pedagogical strategies; and 

• Educational goals 

Each of these components requires a set of data for their configuration, while their suitable combination using an 

intelligent machine produces a final outcome that is the teaching model. 

In the Table 1 we see the components of the teaching model properly combined with their features. 

Components Features 

Knowledge and skills level 

Student’s model 

Student’s needs and motivation 

Personal Learning Styles 

Pedagogical strategies 

Educational goals 

Learning Theories 

Teaching strategies 

Specific Educational goals 

General Educational goals 

Table 1. The components of the teaching model 

At the next stage, we import new stereotypes along with the semantics and the modalities of their use (Tables 2, 

3, 4). Furthermore, we assign preconditions and post conditions so that it is clear how and when these 

components are used. 

Name Student’s model 

Preconditions Profiling mechanism – Rule-based extraction of 

features 

Student’s model 

Knowledge level 

Skills 

Learning style 

Post conditions Personal learning profile 

Table 2. The Stereotype of the student’s model 

Describe how the student learns optimally and what 

his/her strengths and weaknesses are. 

Name Pedagogical strategies 

Preconditions Knowledge-based pedagogical theories and rulebased 

extraction mechanism 

Pedagogical strategy 

Learning Theory 

Teaching strategy 

Combine learning theories with teaching strategies 

and the data extracted from the repository. 

Post conditions Pedagogical strategies adapted to a specific student 

and a specific learning subject 

Table 3. The Stereotype of Pedagogical strategies 

46

Name Educational goals 

Preconditions Course subject, scientific 

domain 

Educational Goals 

General 

Specific 

Describe educational goals determined by the 

educator based on the precondition (i.e. Course 

subject, scientific domain) 

Post 

conditions Set of educational goals for a specific course 

Table 4. The Stereotype of Educational goals 

Using 

the newly added stereotypes 

Most 

problems during the application of the use of UML (located) are in the correct use of symbols and the 

adequate determination of the way of the symbols are combined, their position in the proper, for each case, type 

of diagram and, generally, their utilization in the process of modeling specialized systems. 

On 

the one hand, Figure 2 based on the teaching model of Figure 1 is the collaborative/package diagram of the 

proposed system which is deduced when we use general forms of UML. On the other hand, Figure 3 is the 

collaborative/package diagram of the system we are proposing and is based on the new stereotypes we added. 

Figure 2. Collaboration diagram with UML 

As we see, Figure 2 depicts the general concept of the process but does not provide all the information required. 

Of course, this is just a package diagram, and we will generate a class and an activity diagram to have the desired 

output. But why should we do so many diagrams if we can have all the information in one diagram simple and 

clear but more case specific? Following this train of reasoning, we keep the same diagram using the newly added 

stereotypes. 

In our preceding proposed diagram (Figure 3), the difference may not be huge but it is essential. On the one 

hand, it gives the opportunity to the developer to combine the components of the system, after having 

comprehended the why and the how of the process designed by educators and software engineers. On the other 

hand, the same diagram facilitates educators and software engineers in communicating their concepts and wishes 

to the developer. Overall, the diagram we are proposing helps developers, educators and software engineers have 

some common ground of communication and collaborate with each other. 

47

Case Study 

Figure 3. Collaboration diagram with new stereotypes 

In the following case study we are presenting an example of new stereotypes of the design of the teaching model 

for the specific purpose, as describe above. 

The scientific domain is Mathematics and addresses students of the 2nd level in Junior High School. The general 

educational goal is the comprehension of the theorem of Pythagoras and the assimilation of the knowledge 

acquired. 

Educational Scenario- Specific Teaching Model 

Preconditions 

Figure 4. Collaboration diagram with new stereotypes: Specific teaching model 

• Discipline: Mathematics 

• Course subject: Theorem of Pythagoras 

• Target group: Students of the 2nd level in Junior High School 

• Level: Α (acquisition of basic knowledge) 

48

• Prerequisites: basic knowledge (i.e. what a triangle is, kinds of a triangle, area of a rectangle) 

• Learning style: Student type A (the individual learning style has come out of previous special tests of 

skills and preferences during which there have been determined 4 basic student types. Student type A: a 

balanced way of learning). 

Conclusion 

The model proposed in this paper identifies the problems of web learning and intends to explore practical 

solutions. Whether our solutions are as effective as we designed then to be will become clear when we evaluate 

the first real web courses, which we will have developed while using the model. 

To sum up, from our presentation we can deduce that by expanding UML towards the design of educational 

modeling we fill in an existing gap in the modeling of such specialized software. Of course, we could have used 

only IMS specifications, but then we could have solved the problem partly. So, we have been making an effort to 

combine UML-which is the best know modeling language in Software Engineering- with IMS specificationswhich 

are widely accepted by specialists in Education and Instructional Design. In this way, the proposed 

solution comes to complete and support the advantages that the use of IMS standards, aiming to provide a 

complete solution in the design and the development of educational software. Thus, on the hand, we have the 

standards of the description of how to manage and transfer the educational material and, on the other hand, we 

know how to formulate their use in the technical part. 

More specifically, as shown in Section 4, we have presented a new set of components which can facilitate the 

design of educational software. The newly added stereotypes offer a specialized view of the prospective software 

from an educational stand, while rendering the modules that software engineers must develop understandable. 

The main aim of our research is to provide educators, specialists in every field and computer scientists with a 

systems that gives them a common ground of understanding so that all these specialists can communicate their 

ideas in spoke and written form and understand each other. 

Thus, we can claim that there are two primary advantages of the system we are proposing in this paper. First, it 

gives the opportunity to the developer to combine the components of the system after having comprehended the 

why and the how of the process designed by educators and software engineers. Second, the same diagram 

enables educators and software engineers to communicate their concepts and wishes to the developer. Overall, 

the diagram facilitates the collaboration and understanding of developers, educators and software engineers. 

Concluding, we are still working on modeling patterns of the whole process of educational software production 

and the expansion of stereotypes, which are based on these patterns. 

References 

Conallen, J. (1999). Modeling Web Application Architectures with UML. Communications Of The ACM, 

42(10), 63-70. 

Lionarakis, A., & Rokou F. (2000). Themes and Problematic for Open and Distance Learning. Athens: 

Propompos (in Modern Greek). 

Rokou, F. (2002). Distance Learning with Hypermedia Technologies: Design pedagogical Model and 

communication process. Athens: Kritiki (in Modern Greek). 

Pantano Rokou, F. (2002). A Pedagogical Model for Distance Education with the Use of Information and 

Communication Technologies. Asian Journal of Information Technology 2(1), 08-12. 

Nanard, J., & Nanard, M. (1995). Hypertext design environments and the hypertext design process. 

Communication oF The ACM, 38 (8), 49-56. 

Hennicker, R., & Koch, N. (2003). A UML-based Methodology for Hypermedia Design. Retrieved October 9, 

2003 from http://citeseer.nj.nec.com/hennicker01umlbased.html. 

49

EDUCAUSE (1999). IMS Learning Resource Meta-data Best Practices and Implementation Guide. Version 

1.2.1. Retrieved October 9, 2003 from http://www.imsglobal.org/metadata/imsmdv1p2p1 

/imsmd_bestv1p2p1.html 

Adelsberger, H., Bick, M., & Pawlowski, J. (2000). The Essen Model. A Step Towards a Standard Model of 

Learning Process. Retrieved October 9, 2003 from http://citeseer.nj.nec.com/update/515384 

IMS Global Learning Consortium. (2003). IMS Learning Design Best Practice and Implementation Guide 

Version 1.0 Final Specification. Retrieved October 9, 2003 from http://www.imsglobal.org 

Naeve, A. (1999). Conceptual navigation and multiple scale narration in a knowledge manifold. Retrieved 

October 9, 2003 from http://cid.nada.kth.se/pdf/cid_52.pdf 

Berner, S., Glinz, M., & Joos, S. (1999). A classification of stereotypes for object-oriented modelling languages. 

In, France, R., & Rumpe, B. (Eds), In Proceedings of the UML’99, Lecture Notes on Computer Science, Berlin: 

Springer-Verlag, 1723, 249-264. 

France, R., Evans, A., Lano, K., & Rumpe, B. (1997). The UML as a Formal Modeling Notation. In Proceedings 

of the OOPSLA'97, Workshop on Object-oriented Behavioral Semantics, Atlanta Georgia, USA, 75-81. 

Booch, G., Rumbaugh, J., & Jacobson, I. (1999). The Unified Modeling Language: A User Guide, area/city: 

Addison Wesley. 

50

Brace-Govan, J., & Gabbott, M. (2004). General Practitioners and Online Continuing Professional Education: Projected 

Understandings. Educational Technology & Society, 7 (1), 51-62. 

General Practitioners and Online Continuing Professional Education: 

Projected Understandings 

Jan Brace-Govan 

Research Fellow 

Department of Marketing 

Monash University 

Faculty of Business and Economics 

P. O. Box 197, Caulfield East, 

Victoria 3145, Australia 

Tel: +61 3 9903 2491 

Fax: +61 3 9903 2900 

jan.brace-govan@buseco.monash.edu.au 

Mark Gabbott 

Head of Department 

Department of Marketing 

Monash University 

Faculty of Business and Economics 

P. O. Box 197, Caulfield East, 

Victoria 3145, Australia 

Tel: +61 3 9903 2547 

Fax: +61 3 9903 2900 

mark.gabbott@buseco.monash.edu.au 

ABSTRACT 

Continuing professional education seems to be particularly suited to the online environment with 

opportunities to communicate anywhere anytime. This appears to be convenient and time efficient for the 

busy working professional. The views of practising professionals were sought and form the basis of this 

paper. Primary care physicians at two locations, who actively pursue continuing education, discuss the role 

of information technology in their daily professional practice and give their views about online education. 

This professional group is a good example of how a particular interaction style is significant to the 

collective cultural mores of a group. The paper concludes that these group preferences need to be taken into 

account when designing continuing education and some suggestions are made for design. 

Keywords 

Online education, Continuing professional education, Continuing medical education, Web-based education, 

General practitioners, Primary care medicine 

Introduction 

The use of information and communication technologies in learning has grown rapidly (Conole, Hall and Smith, 

2002). Online or web-based education has been used in a variety of contexts and across the different stages of 

education, including professional development (Brink, Munro and Osborne, 2002; DeLacey and Leonard, 2002; 

Lockwood, 2001; Mason, 1998). Web-based education offers flexible access and communication where the 

potential to form personal and professional networks electronically is significant. Networked continuing 

professional education (CPE) is argued to have many advantages (Bernath and Rubin, 2001; Kirkpatrick and 

McLaughlan, 2000), including those of access and flexibility, which would appear to be especially useful to 

busy, working professionals (Conole et al., 2002). With CPE available anytime, and almost anywhere, webbased 

learning for professional development has much to offer continuing self-directed professional education. 

However, web-based CPE is a comparatively recent development (Anderson and Kanuka, 1997; Brosnan and 

Burgess, 2003; Friedman et al., 2002; Peterson, 1999) and can be described as an innovative product (Lally, 

2002). This paper will consider the views of potential users of a particular type of web-based CPE, physicians in 

general practice in Australia. 






51

Web-Based Education as Innovation and Continuing Medical Education 

The underlying principle that informs this paper is that taking the culture of the target group into account will go 

a long way towards overcoming any potential reluctance to adopt technology, as well as encouraging 

engagement with technology. Innovative products must be “positioned in the marketplace in ways that match the 

understandings and expectations of customers” (Lally, 2002: 119). These customer understandings are based on 

what consumers already know, and novel products rely on consumers being able to speculate and project ideas 

about the nature of the new experience. In line with this, online education evaluators have established that 

participant perceptions in CPE are an important part of adoption (Anderson and Kanuka, 1997). The uptake of 

technology is a complex matter and human responses often appear paradoxical (Mick and Fournier, 1998). It has 

been proposed that to “ignore, refuse and delay adoption of technology are arguably quite judicious choices” 

(Mick and Fournier, 1998: 141) and that this can protect the unwary consumer from expensive and timeconsuming 

mistakes. Therefore, while attitudes towards technology can change over time (Lee et al., 2002), the 

issues surrounding the human interface with IT need to be given close consideration if novel advances are to be 

widely adopted. In particular, it makes good sense to take into account the specific needs of the targeted users 

and to investigate related understandings and expectations. 

Part of the context for the targeted users in this instance includes a specific type of work organisation. IT in any 

workplace is complex but in the context of general medical practice arguably some of the paradoxes of 

technology are more visible. Firstly there is the issue of the reliability of novel technology, particularly in 

primary care practices where the kinds of support given in larger organisations is expensive and not available as 

easily, nor as quickly. Then there is the time-consuming nature of learning new software, which can be a serious 

deterrent to a self-employed professional. While Internet and IT compliment rather than displace other activities, 

they are also malleable and changing fast (DiMaggio et al., 2001). These rapid changes place demands on small 

enterprises and therefore this is an area that requires ongoing and context specific research. 

A further aspect of understanding that needs to be included here is the professional’s need to learn in, and for, a 

specific context (Eraut, 1994) and the different learning needs that arise as their expertise develops (Daley, 

1999). Conceptualisations of professional competence recognise the significance of continuing education, or 

lifelong learning, as an integral part of the complexity of professional knowledge (Eraut, 1994). Indeed 

maintaining a zest for learning was argued to be one of the defining features of professional work (Houle, 1980), 

with continual learning that goes beyond simple up-dating a crucial element in professional competence 

(Cervero, 1988). Continuing medical education (CME) has a long history (600 years) but it is a relatively recent 

development to have this available on the Internet (Peterson, 1999). A recent assessment of CME online found 

205 host computers, 80 of which were American medical schools and 55 of which were commercial sites 

(Peterson, 1999: 243). These figures suggest that the Internet is becoming a significant, and recent source of 

CME. 

If web-based CME is taken to be an innovation, then Lally's (2002) suggestion is that investigating potential 

users’ experience of related activities and asking them to speculate about the novel product could enhance 

understandings about potential participants’ expectations. Moreover, although the positive position for online 

education is well supported in the literature from several different standpoints, these tend to be written from the 

perspective of the developer, the teaching institution or the evaluator. It is also important to take the user's 

perspective into account. So, what is the view of practising professionals, who make regular and consistent use 

of continuing education? How do they perceive web-based education? A good understanding of the target group 

supports a user-centred approach to design (Lee et al., 2002; Spector and Wang, 2002) where varying views of 

users can be informative (Clulow and Brace-Govan, 2003). It was suggested that targeting online education to 

the needs and attitudes of specific user groups enhances, at least initially, the likelihood of uptake and 

encourages a positive perspective. The overarching suggestion here is, that it is beneficial to actively work with 

particular groups to create the continuing education that they need for their context (Daley, 2000:40), not only in 

terms of content, but also in terms of engagement and interaction preferences. 

With regards to the question raised, general medical practitioners, or primary care physicians (GPs), are in quite 

a particular and interesting position. As a professional group they have a history of using professional 

development and in Australia (like Canada) they receive accreditation points for continuing medical education 

(Cervero, 2000). At the same time, they are required to deal with the rapid changes in medical knowledge and 

implement these in a practical, face to face context with their patients. In a review article that defines and 

assesses what professional competence might mean to the medical profession, Epstein and Hundert include the 

self-directed acquisition of new knowledge, being able to recognise gaps in their own knowledge and using 

appropriate resources as important factors in the continuing professional education of competent doctors 

52

(2002:227). Professional work has been noted as undergoing change in recent times (Wilson, 2000). Some argue 

that GPs have maintained more autonomy than might be indicated by suggestions of either de-professionalisation 

or proletarianisation of the professions (Lewis, Marjoribanks and Pirotta, 2003). Others suggest that managerial 

and professional work are converging in an elite strata of neo-entrepreneurialism (Leicht and Fennel, 2001). 

What these discussions underscore is the widely recognised need for professionals, including GPs, to undertake 

continuing education to maintain currency, competency and competitiveness (Curry and Wergin, 1993; Harris, 

1993; Leicht and Fennell, 2001; Lewis et al., 2003; Mott and Daley, 2000). But, would online delivery of CME 

for GPs address their needs effectively and efficiently? A first step would be to find out how GPs perceive 

information technology (IT) and its place in their professional practice. This may draw attention to any issues 

that might need to be considered when introducing online CME. 

There are quite divergent views on the usefulness of information technology to general practice. Some claim that 

there is "little evidence that computers are of any value for healthcare in general, especially general practice" 

(Bolton et al., 1999: 962) while others suggest that information technologies play an "enabling role" and describe 

doctors as knowledge workers (Wickranasinghe and Lamb, 2002). A review of studies of computers in primary 

care locations found that overall general practitioners were positive about information technology but had five 

reservations (Mitchell and Sullivan, 2001). GPs expressed concerns about the cost of computerisation, the time it 

took to master the processes, the lack of adequate training, the impact that engaging with a computer might have 

on their interactions with patients and, the potential loss of confidentiality in record keeping (Mitchell and 

Sullivan, 2001:281). 

Reports commissioned to investigate general practice and IT in Australia suggest that IT has much to offer GPs 

in terms of streamlining data management and supporting patient management (GPCG, 2001; Richards et al., 

1999). Although IT uptake amongst Australian GPs has been described as slow (GPCG, 2001:6; Richards et al., 

1999:4), or “in its relative infancy” (Health Online, 2001:6), nonetheless others argue that government incentives 

to make use of IT in the day to day management of patients and practices have had an impact (Western et al., 

2001: 4). A recent national representative survey reported computerisation of general practice at around 90%, 

and therefore widespread. It also noted that practices with over three doctors were, not only considered to be 

large, but they were also more likely to be computerised (Western et al., 2001:43). GPs did not report feeling 

anxious about computer use or learning computer skills, including doctors who were older (Western et al., 2001: 

101). However, GPs found learning new software time-consuming (Western et al., 2001: 102; Mitchell and 

Sullivan, 2001:281) and, remained wary about the amount interference that computers introduce to their 

relationship with the patient (Western et al., 2001:101; Mitchell and Sullivan, 2001) seeing writing as less 

invasive than typing (ACNeilson, 1998:34). 

This paper reports on an exploration of how IT was used and perceived by GPs with a view to understanding 

how IT could be used in CME. Two medical practices with access to IT and an active approach to CME were 

invited to explore some issues with the author. These investigations at the local general medical practice level 

show that continuing professional education, construed by the GPs as CME, and online delivery are not quite as 

straight forward as the literature suggests. 

Method 

While there is some work on GPs attitudes to IT, this particular area of IT involvement with CME is underresearched, 

so an exploratory, qualitative approach was adopted (Patton, 1990; Strauss and Corbin, 1998). Two 

metropolitan general medical practices met the research criteria, were approached and agreed to take part in the 

study. 

Research Criterion Justification for Criterion 

1. Practice with more than three doctors Larger practices more likely to be computerised 

2. Metropolitan practices Improves access to wider range of CME 

3. Access to IT at work Good IT connections enhances usability 

4. Similar location and patient access Focus comparison on research issues 

The first research criterion was that the practice should have more than three doctors because larger practices 

were noted as more likely to be computerised. Secondly, using metropolitan practices meant that the doctors all 

had reasonable access to CME of various kinds, which is not always the case for rural practitioners. At the time 

of the interviews, general practitioners were required to gather a certain number of CME points every three 

years, or triennium, in order to practice. CME points could be gathered from a wide variety of CME activities 

53

ut a focus on city-based doctors enhanced the likelihood of personal preferences being available as an option. A 

third element the research required was that the practices had good access to IT. This was the case here and both 

practices provided each GP with a computer and had cable network installed. Fourthly, the practices also needed 

some other structural similarities in terms of geographic location and patient access to narrow the focus of the 

comparisons. They were both located in suburban Melbourne, in well kept, converted residential properties, 

conveniently located on major roads with access to parking in nearby side streets or public transport. They were 

both open five and a half days a week and later field observations noted that both practices were busy. With 

more than three GPs in each practice, these practices were above average size for Australia (Richards et al., 

1999). Most Australian general practices (60%) have only one or two doctors, 26% have three to five doctors 

and only 14% have six or more GPs (Western et al., 2001:29). However, 43% of practices in capital cities would 

have three to eight doctors (Western et al., 2001:31). In addition, 96% of practices with three or more doctors use 

computers (Western et al., 2001:31). Therefore, these two practices would be considered large by Australian 

standards, but not unusual for a city location and typical in that they are computerised. It was assumed that the 

number of doctors also gave rise to a need to communicate with each other about practice management and 

patient care. One significant point of difference between the two practices was maintained. Some general 

practices bill the government directly through a national health scheme and the patients are charged nothing 

directly. Other practices charge fees and the patient must seek a partial rebate through the health insurance 

scheme. Practice A charged fees for consultations and GPs usually saw around four patients an hour. This 

practice also had more GPs who worked part-time. Practice B relied mostly on bulk billing through the national 

health insurance scheme (Medicare) and GPs took around six consultations an hour. 

The principle data gathering instrument was the semi-structured interview schedule, which allowed the GP 

participants to explore the concepts of CME and IT both separately and together from within the context of their 

own medical practice. As well as the discussions with the GPs in the practice, observations about the practice 

were noted in a field diary and documents used to communicate with patients were also entered as data. These 

sources were used to corroberate information given by the interviewees. Furthermore, one participant from each 

practice was interviewed about the practical aspects of IT such as available hardware, installed software, Internet 

connections and IT support. In one practice this was a GP and in the other practice this was the Practice 

Manager. In all, 10 participants were interviewed: nine GPs (five from practice A and four from practice B) and 

one practice manager. A doctor that was a director of the practice, as well as a doctor that was a partner in the 

practice, were interviewed in both locations. The remaining five doctors were employed by their respective 

practices. There were two GPs in each practice who did not consent to be interviewed. The participants were 

ages 30 and older, with half of them in their 30s. Two participants were men, and eight were women. Three 

doctors worked full time and six doctors worked part time. In this paper individuals will be quoted using a 

number system for reasons of confidentiality. 

One further source of information was utilised by the study. The Australian Government’s Department of Health 

and Aged Care through the General Practice Computing Group had developed resources to support general 

practitioners on issues surrounding IT and general practice. A web site of case studies was available on the 

Internet (IM in GP Lead Practices, 2002). In all, six case studies described the experiences of exemplar, lead 

practices and acted as “peer education models” (IM in GP Lead Practices, 2002: 1). The lead practices used 

included a mix of metropolitan and rural practices and varying numbers of doctors with four cases have more 

than three doctors. The one constant factor across the cases was a computer on every doctor's desk with access to 

email and Internet. This kind of benchmarked, or exemplary performance is used to good effect in medical 

education (Mazamanian and Davis, 2002: 1058). In this instance, the descriptions present a positive perspective 

on the adoption and utilisation of IT. However, with this bias in mind, comparisons between the research 

interview data from this exploratory study and the descriptions from the lead practice web site were useful for 

further verification and context. It is noteworthy that the experiences described cover patient and practice 

management but do not include web-based CME. 

The research question was divided into three separate sections. Firstly, the doctors were asked to describe the 

CME activities that they had attended in the previous twelve months. Secondly they described the extent to 

which they incorporated IT into their daily work practices and how they felt about IT. Finally the participants 

were asked to address the main question, how do you think you would feel about taking a web-based CME 

activity and what issues would you consider relevant to that speculation? The paper will build a description 

following the same progression. The GPs' engagement with CME and use of IT will be described, followed by 

an analysis of the GPs perception of using IT to access CME. The paper will close with the implications the 

analysis has for the implementation of CME online. 

54

CME Activities 

These GPs were busy professionals with full appointment books who were concerned to maintain the currency of 

their own professional knowledge in order to best serve their patients and ensured that they conveyed relevant 

professional knowledge to the others in their practice. There was a range of CME available to them and all the 

doctors were actively engaged in some kind of continuing education on a regular basis, as can been seen from 

table 1. 

CME Activities Attended in 12 months No. GPs* 

Lecture Series at Local Hospital 5 

Conference at Hospital 2 

Short Course 1 

Alternative Medicine Course 1 

Formal Journal Reading Group 5 

Regular Discussion Group 4 

Drug Co. Sponsored Lectures 1 

* Some doctors attended more than one type of CME 

Table 1. CME Activities Attended by GPs in Previous 12 months 

The most popular option regularly attended by five GPs was a series of weekly lectures arranged by the local 

hospital on varying topics. The presentations were usually made by specialists from within the hospital. A 

further two GPs reported recent attendance at a conference at the central Children’s Hospital on paediatrics and 

one other doctor was taking a short course on counseling skills. All of these activities counted as CME points for 

their ongoing accreditation as general practitioners, although they also all explained that they were well past the 

minimum points for a triennium. The remaining GP was attending a course on an alternative medicine and could 

not gather CME points through this educational activity. However, this doctor gathered the points needed for 

accreditation through journal reading and attending a journal club, or discussion group, which met regularly at 

the practice. Although both practices held regular meetings to exchange information about patients and 

administration, one practice had a more formalised approach. In this journal club the GPs made presentations to 

pass on knowledge from their other CME activities, or invited a speaker to present to them on a specific topic. 

Another option for education which accumulated CME points were lectures sponsored by the drug companies 

which the GPs were prepared to attend but they were more wary of the information presented here than the other 

options they used regularly. 

In short, all these GPs were actively, and regularly, engaged in CME and, although this is a requirement for their 

accreditation, their engagement with this was more a question of being a good quality professional practitioner. 

As this GP said: 

I’d like to maintain a certain level of practice and I think you’ve got to continue with education too. 

[doctor 3: p13] 

A view given in more detail by this GP: 

From my point of view, I’ve been doing it a long time, and it’s nice to just get your thoughts re-ordered, 

get them into perspective, …. Next time you see that problem you deal with it much quicker, and more 

confidently because you know what you are going to do next. 


In addition, although the format varied between the two practices, there was some sharing of information about 

patients, practice management and the content of continuing education. Overall the GPs were active in their 

pursuit of their own self-directed learning and made clear links between CME and professional competence. The 

next step was to ascertain how they perceived IT in their day to day practice and to make some assessment of 

this. It was noted earlier that the Lead Practice web site described achievements of six case study practices. 

These descriptions will be taken to be benchmarks, as the site suggests. A comparison between the practices in 

this study and the exemplars on the Lead Practice web site gives an indication of the extent to which IT is used 

and the extent to which it could be used. 

55

IT Use and Attitudes 

All the GPs used IT in the course of their everyday practice, although some used IT with more enthusiasm than 

others. A policy decision had been taken in both practices that IT would be made available to all the GPs, but 

neither practice would do more than encourage GPs to make use of the system. In each practice one person was 

described as being the in situ 'expert'. In one practice it was the Practice Manager and in the other it was the 

Practice Director. Both were particularly positive about the usefulness of IT in managing the practice in terms of 

appointments, staff rosters, billing and banking and both practices used IT for practice management and patient 

management. The use of IT in patient consultations and patient histories was a matter for individual GPs. All the 

GPs used IT for prescriptions, pathology requests and reports, and letters to specialists. The software that both 

practices used also automatically indicated when drugs were incompatible and gave patient summaries. 

This had some resonance with the Lead Practice Case Studies provided on the Australian Division of General 

Practice web site (IM in GP Lead Practices, 2002). At this site six cases are presented as exemplars in the use of 

IT in general practices. Comparing the Lead Practice Cases with the practices from this study, IT was reported as 

being used for similar purposes: pathology, patient databases, rosters, billing, appointments and other practice 

management tasks. Generally the GPs in this study recognised the functionality of IT echoing many of the 

comments at the Lead Practice web site. For example printed scripts increased readability for pharmacists 

(doctors' writing being widely perceived to be illegible). IT was also useful in the case of multiple medications 

or allergies by giving automated warnings. In the practice that relied on Medicare bulk billing, the electronic 

summaries were noted by all these GPs as quicker and easier to "flick through" than written patient histories. 

However, neither practice was prepared to go to paperless, or rely entirely on electronic record keeping and this 

was an important point of difference with the web site examples. In the Lead Practice Cases there was positive 

reporting about using electronic patient records. In the practices in this project, the GPs were reluctant and 

resistant to the idea of only having electronic records. There were three issues here: reliability, accuracy and 

confidentiality. 

In the first instance it was the reliability, of the technology, or rather the lack of it. A GP described the effect: 

We’ve had a couple of instances when the computer’s gone down here and it’s just thrown the whole 

surgery, the whole practice, into chaos. 


And at the other practice a GP said: 

I just don’t trust IT enough to go to a full paperless office. 


Apart from the impact that was felt when the system was not operational, not only was it difficult to work when 

the computer system was not functioning, but there was also a potential to lose important patient information. 

This then raised the issue of accuracy in record keeping, not only for appropriate patient care, but also for 

professional responsibilities to other agencies that might require, or rely on, accurate patient histories, for 

example in insurance claims. The accuracy here related to maintaining full and complete records of patient visits. 

The concept of accuracy was further extended by one GP to include their need to use drawings in their notes in 

order to fully record details of a case [doctor 8: p4], which was not possible in the medical software available. In 

addition there were also deep concerns for the confidentiality of records. They worried about the implications for 

patient privacy in electronic record keeping. These concerns ranged across what would be on screen and visible 

to the patient, the impact of concentrating on typing rather than on the patient, and who would have access to the 

patient’s information. Concerns that were also noted by Mitchell and Sullivan (2001). The reluctance to maintain 

electronic patient records had the further effect of viewing IT as a source of "doubling up" on record keeping. 

The GPs’ perception was that to overcome the lack of system reliability, to maintain accuracy and to address the 

need for patient confidentiality, they would have to maintain two sets of records, one paper based and one 

electronic. This contrasts with the efficiencies of IT promised on the Lead Practice web site. 

Although all the GPs used IT for managing information about the practice and their patients, only one GP from 

each practice could be described as fully in favour of IT (see table 2). 

56

GPs’ attitudes towards IT use in 

General Practice 

No. GPs 

Very positive 2 

Neither positive nor negative 4 

Somewhat negative 2 

Very negative 1 

Total 9 

Table 2. Summary of GPs’ Attitudes Towards IT Use in General Practice 

The remaining seven GPs had varied reactions: two GPs in each practice were equivocal about IT (it was OK but 

it had limitations); one GP in each practice was not particularly happy about using IT; leaving one GP who 

described themselves as "a bit anti-computer" and thought that generally the community would be happier 

without computers [doctor 6: p5]. 

Individual attitudes to the computer were further revealed through the GPs’ use of the Internet to obtain 

supporting information for their professional practice. Although attitudes varied about the potential usefulness of 

the Internet, only two of the GPs would actually use the Internet to find information for themselves on a regular 

basis [doctor 7, doctor 9] and another GP noted the helpfulness of visual material from the Internet in 

understanding some medical conditions [doctor 2: p11]. The remaining six doctors would direct patients to 

useful and reliable sites if they knew of one giving the kind of information they would support. Overall though, 

the GPs were troubled by the inaccuracies of medical information on the Internet and were concerned about the 

effect that this could have on patients, potentially causing unnecessary anxiety. This more ambivalent attitude 

towards IT was an important point of difference to the reported cases on the Lead Practice where such doubts 

were not expressed. In this study, even those GPs who were kindly disposed towards IT were concerned about 

the patients’ ability to assess medical information. Furthermore, in their discussions about the Internet and 

patients two key, linked issues emerged. Firstly the GPs expressed the need for reliability of medical information 

but a sense of unreliability with Internet sourced information. Secondly though, reliability is often established 

through the reputation of the information provider. The link between the reliability of information and the 

reputation of the provider of information becomes even more important in the provision of CME. 

Perceptions of web-based CME 

Considering the GPs attitude towards IT in their general practice (see table 2), it is unsurprising that these 

doctors were not especially in favour of using IT for their CME. However, the position was not simply a 

negative perspective on computers. GPs used their continuing education for more than ensuring good quality 

professional practice and their complex motivations had a bearing on their perspective on using IT for CME. It 

has been recognised for some time now that lectures organised by local hospitals for general practitioners not 

only offer advice and information, but that they are also a useful mechanism to encourage GPs to refer their 

patients to the hospital’s specialists (Cervero, 2000; Daley, 2000). However this works both ways, because for 

the GPs these lectures offered them the opportunity to evaluate the specialist doctors available and assess for 

themselves the quality and depth of the specialists' expertise, as well as their suitability for the GPs’ patients. 

Several of the GPs [doctors 2, 4, 5, 8, and 9] specifically stated that their choice of lectures depended on the 

reputation of the speaker. This professional network relied on trust and face to face interaction with implications 

for the GPs own patients. The referral process was an important reflection of the GPs themselves, as this GP 

explained: 

I think you can judge a lot from when you meet somebody, what sort of person they are and how you 

think the patients might interact with them, because I think the patients, from the referral point of view, 

the patients – it reflects on me, how a patient finds the particular doctor they end up seeing. 


The judgement described by this GP was a significant area of professional skill for general practitioners. One GP 

[doctor 10: p5] used the term "artistry" to convey the extent to which practical medical knowledge was used 

interpretively when consulting with a patient. Through their professional practice and the ongoing experience of 

being a GP, doctors were immersed in the day to day experience of knowing, of creating understandings for 

themselves and others through the face to face interaction that was the bedrock of their general practice. Eraut 

(1994) asserts that professional knowledge needs to be characterised through the context in which it is used. In 

the context of general practice, GPs' skills relied on the extensive use of face to face interactions. It was the 

mode of communication that characterised their professional activity, as compared to a radiologist or a surgeon 

57

for example. Therefore, for GPs to feel the need to establish someone's professional reputation through face to 

face interaction and to maintain their professional network in this manner was to be expected. This interaction 

process gave them the opportunity to assess specialists for referrals in a manner in which the GPs felt 

accomplished. It also meant that additional related topics could be touched on and in this way diversity with 

relevance was accessed in a succinct and time effective manner. One of the GPs who was in favour of IT, and 

used IT quite extensively for information gathering, said this: 

Online you are probably going to look up a specific product – not a product, maybe a specific problem. 

Whereas you might look up a kidney problem but with the last talk at {the local hospital} we probably 

covered 30 kidney problems. ….. {there’s} question and answer, and you can interrupt them during the 

talk [doctor 7: p8] 

Asked about conducting this kind of educational or information and networking activity online, the response was 

that this would be too much reading, and too time-consuming. The GPs described a strong perception that 

computers were only for reading and not especially different to journals. Furthermore the process of typing 

responses, as compared to talking, was also described as time-consuming. Face to face interaction with other 

people was seen not only as time efficient, but it was the additional ingredient offered by lectures and perceived 

(however erroneously) to be missing from IT facilitated interactions. The GPs expressed a need for interaction in 

the education process: 

But for education it’s really important to actually have interaction. So, it [online education] is really just 

another reading mechanism. [doctor 5: p5] 

It [online education] would be akin to reading the dictionary to me. [doctor 4: p15] 

I read journal articles every night and I go to sleep. ….. if you’re just reading it and it’s not being 

presented in an interesting way, it’s all the same. [doctor 2: p4] 

Another GP [doctor 9: p6] who was also active in gathering information from online resources also preferred to 

go to presentations but the reasoning here focussed not only on accurately grasping information but also, 

importantly, the reputation of the source. The reputation, the trustworthiness and accuracy of the information, 

and the need to understand the perspective of the source was a concern for the doctors. The doctors preferred 

method of establishing reliability and reputation was through face-to-face interaction and talking. 

One last issue connected with reputation that needs consideration is the link between software and advertising by 

drug companies. Unsolicited advertising was an integral part of the patient management software that the 

practices used. For one GP this advertising further undermined the computer as a reputable source of 

information. The link here relied on views about drug companies. The lectures that the companies provided 

(usually with lunch or dinner) which earned GPs CME accreditation points, were widely seen as presenting 

information that was potentially useful but usually slanted in the drug companies’ favour. It was information that 

needed to be dealt with circumspectly because the information provider could not be seen to be fully impartial 

and objective. When this GP was asked about online education the association between software and unsolicited 

drug advertising was made immediately. It was this GP’s perception that medical practice software was simply 

an easy avenue through which drug companies could sell their products. This perception distorted the GP's view 

of online education and, although an extreme view, it does express the reservations that many GPs feel about 

drug companies and biased information. A similar point was raised by Peterson (1999) around the quality of 

commercial provision of CME on the Internet. Peterson questions the impact of any underlying market interests 

and the potential for imbalance in provision due to specific sponsor needs (1999:247). The doctor’s comment in 

this study about advertising in software reinforces the point that GPs put a high value on the interpersonal, 

professional network on which they relied for quality information. The question that arises now is what do these 

collected attitudes imply for online CME? 

Discussion 

Although selected purposefully to explore the issues of CME and IT with GPs who had good access to both, the 

limitations of this study due to the small sample size are acknowledged. Therefore, it is not possible to generalise 

from the results and further research is required. However, the motivations and perceptions of learners are 

crucial (Brink et al., 2002; Clulow and Brace-Govan, 2003), particularly in regard to their views on technology 

(Lee et al., 2002), so some implications can be derived from this research. The literature showed a wide range of 

views on IT in general practice (Bolton et al., 1999: Wickranasinghe and Lamb, 2002) and the GPs in this study 

58

also expressed such diversity. Principally the concerns identified by Mitchell and Sullivan (2001) regarding 

costs, time management, lack of training, impact on patient relationship and potential loss of confidentiality for 

patient records were also expressed here. IT was perceived by the GPs in this study as a functional, useful tool 

for practice and patient management but they were reluctant to rely solely on IT and to surrender their paper 

records. In addition, as suggested by Epstein and Hundert (2002), the GPs in this study actively and critically 

pursued self-directed learning. However, the GPs perceived online, or web-based, CME as lacking interaction 

and no different to their journal reading. Furthermore, reading was seen as more time-consuming and less 

informative than attending presentations given by experts. It was suggested that face to face interaction was a 

key skill for their profession and integral to the process of establishing the credence of other professionals. It was 

a network of trust, a community of practice that was built using the face to face evaluation of people through the 

use of the GPs' well-tuned skills of gauging individuals on the basis of their self-presentation and interaction. In 

short, CME was a time efficient means to gather not only needed information but also a means to extend and 

verify a professional network, but computers were tools like books and journals not for interactive 

communication. 

This begs the question, how then can GPs be persuaded to view online CME positively? What kinds of issues 

need to be considered when designing web-based continuing education for this particular professional group? 

Perhaps the questions raised by Cervero (2000) on CPE in general need to be paraphrased for online CME and 

addressed first. Why is CME going online? Who will benefit from this? Who will provide the online content? 

Drug companies, Universities, professional associations or collaborations? These issues need to be addressed and 

communicated to the wider GP community. Currently, much of the advice about online delivery of education 

relies on self-directed learning being part of a large organisation's strategy for staff development and there is 

little research focusing on small enterprises (Brink et al., 2002). Most general practices would be classified as 

very small enterprises with below twenty staff. Furthermore, GPs, like many other professionals in small 

professional service firms, take responsibility for their continuing education individually (Mazmanian and Davis, 

2002). Therefore, as a first step the advantages of asynchronous flexibility needs to be more widely promoted to 

individual GPs, along with the benefits of developing an electronic professional network to extend the 

boundaries of trustworthy and informative colleagues. 

It was also clear from the results reported here that the reputation and relevance of any online presenter is 

important. So a second step would include establishing the presenter's credentials. In the instance of the GPs, 

perhaps this is the time to clarify any role or input from drug companies. In addition, there was an expressed 

need to interact with that expert presenter (Mazmanian and Davis, 2002). One potentially useful model would be 

the Virtual Seminar in Distance Education where professional development was team taught to specific 

geographic locations (Bernath and Rubin, 2000). A key element of this model was the inclusion of a highly 

regarded expert for each of the four areas of theory covered. The expert was available as a source of information 

and motivation. As a third step, topics need to be focused on the specific needs of identifiable general 

practitioners and dealt with in a succinct and informative manner. An option here would be the negotiated group 

learning described by McConnell (2000) where participants took an active role in deciding the direction of their 

co-operative small group work. A further option would be to combine both these suggestions taking care to keep 

time-consuming reading and typing to a minimum. Another design approach that integrated online and face to 

face classes was used successfully at Harvard (DeLacey and Leonard, 2002). Here expert presenters took face to 

face classes and then learners worked on a joint project online. This kind of interaction has been shown to be 

valuable in medical education (Mazmanian and Davis, 2002: 1058). In addition, a crucial element of success in 

the Harvard project was the care taken to fully inform and prepare these learners for the online environment. A 

point that has been made for courses delivered entirely online with no co-presence at all (Brace-Govan et al., 

2001; Clulow and Brace-Govan, 2003). However, co-present communication has been noted as essential for 

other professional groups in project work and management (Bjorkegren and Rapp, 1999; DeLacey and Leonard, 

2002) and it was clearly important for the professional group studied here. The preferences groups have for 

particular kinds of interaction can be usefully incorporated into the overall management of CME online. 

Conclusion 

The study relied on the assumption that consumer perceptions of innovative products derives in part from related 

activities. A further assumption was that understanding these perceptions can usefully inform ways to tailor 

innovative products so that they are more appealing in the first instance. The relevant related activities were the 

GPs’ recent CME and the ways in which they used IT in their medical practice. From this point the GPs were 

invited to speculate on how they perceived web-based CME and, in general, their perceptions were quite 

59

negative. It was suggested that this could be overcome by suitable tailoring of web-based education which would 

need to take into account the GPs’ need for reliability of information and their desire for face to face networking. 

References 

ACNeilson. (1998). Attitudes towards information technology in Australian general practice, Volume 2, 

Qualitative Research Report, North Sydney: ACNeilson, retrieved October3, 2003 from 

http://www.health.gov.au/pubs/gpit/gpit2.pdf 

Anderson, T. & Kanuka, H. (1997). On-line forums: New platforms for professional development and group 

collaboration, Journal of Computer Mediated Communication, 3 (3), 14 pages, retrieved October 3, 2003, from 

http://www.ascusc.org/jcmc/vol3/issue3/anderson.html 

Bernath, U. & Rubin, E. (2001). Professional development in distance education – A successful experiment and 

future directions. In Lockwood, F. & Gooley, A. (Eds.) Innovation in open and distance learning: successful 

development of online and Web-based learning, London: Kogan Page, 213-223. 

Bjorkegren, C. & Rapp, B. (1999). Learning and knowledge management: A theoretical framework for learning 

in flexible organisations. In Jackson, P. J. (Ed.) Virtual working: social and organisational dynamics, London: 

Routledge, 157-176. 

Bolton, P., Douglas K., Booth B., & Miller G. (1999). A relationship between computerisation and quality in 

general practice. Australian Family Physician, 28 (9), 962-965. 

Brace-Govan, J., Wagstaff, P., & Luxton, S. (2001). Orientation module to increase ease of access to online 

discussion forum for postgraduate students. In Kennedy, G., Keppell, M., McNaught, C. & Petrovic, T. (Eds.), 

Meeting at the Crossroads, Australian Society for Computers in Learning in Tertiary Education Conference 

(ASCILITE), Melbourne 9-12 December, 5-9. 

Brink, B., Munro, J., & Osborne, M. (2002). Online learning technology in an SME work-based setting. 


Brosnan, K. & Burgess, R. C. (2003). Web based continuing professional development – A learning architecture 

approach. Journal of Workplace Learning, 15, (1), 24-33. 

Cervero, R. M. (2000). Trends and issues in continuing professional education. In Mott, V. W. & Daley, B. J. 

(Eds.) Charting a course for continuing professional education: Reframing professional practice, 86, San 

Francisco, Jossey-Bass Publishers, 3-12. 

Cervero, R. M. (1988). Effective continuing education for professionals, San Francisco Jossey-Bass Publishers. 

Clulow, V. & Brace-Govan, J., (2003). Web-based learning: Experience-based research. In Aggarwal, A. (Ed.) 

Web-based education: Learning from experience, Hershey, PA: Information Science Publishing, 49-70. 

Conole, G., Hall, M. & Smith, S. (2002). An evaluation of an online course for medical practitioners. 


Curry, L. & Wergin, J. F. (Eds.) (1993). Educating professionals: Responding to new expectations for 

competence and accountability, San Francisco: Jossey-Bass Publishers. 

Daley, B. J. (2000). Learning in professional practice. In Mott, V. W. & Daley, B. J. (Eds.) Charting a course for 

continuing professional education: Reframing professional practice, 86, San Francisco, Jossey-Bass Publishers, 

33-42. 

Daley, B. J. (1999). Novice to expert: An exploration of how professionals learn. Adult Education Quarterly, 49 

(4), 133-144. 

DeLacey, B. J., & Leonard, D. A. (2002). Case study on technology and distance in education at the Harvard 

Business School. Educational Technology & Society, 5 (2), 13-28. 

60

DiMaggio, P, Hargittai, E., Neuman, W. R., & Robinson, J. P., (2001). Social implications of the Internet, 

Annual Review of Sociology, 27, 307-336. 

Epstein, R. M. & Hundert, E. M. (2002). Defining and assessing professional competence. Journal of American 

Medical Association, 287 (2), 226-235. 

Eraut, M. (1994). Developing professional knowledge and competence, London: The Falmer Press. 

Friedman, A., Watts, D., Croston, J. & Durkin, C. (2002). Evaluating online CPD using educational criteria 

derived from the experiential learning cycle. British Journal of Educational Technology, 33 (4), 367-378. 

GPCP (General Practice Computing Group), (2001). General Practice Computing Group Strategic Framework 

for IM & IT in General Practice 2001-2005, retrieved October 3, 2003 from 

http://www.gpcg.org/aboutgpcg/keydocs.html 

Harris, I. (1993). New expectations for professional competence. In Curry, L. & Wergin, J. F. (Eds.) Educating 

professionals: Responding to new expectations for competence and accountability, San Francisco: Jossey-Bass 

Publishers, 17-52. 

Health Online. (1999). Health online: A health information action plan for Australia. Canberra: Department of 

Health and Aged Care. 

Houle, C. O. 1980. Continuing learning in the professions. San Francisco: Jossey-Bass Publishers. 

IM in GP Lead Practices. (2002). IM Lead Practices, iM-bulletin edition 33, May 9 th , retrieved October 3, 2003 

from http://www.adgp.com.au/site/index.cfm?display=375 

Kirkpatrick, D. & McLaughlan, R. (2000). Flexible lifelong learning in professional education. Educational 

Technology & Society, 3(1), 24-31. 

Lally, E. (2002). Consuming home technology: Consuming home computers. In Miles, S., Anderson, A. & 

Meethan, K. (Eds.) The changing consumer: Markets and meanings. London: Routledge, 117-130. 

Lee, J-S., Cho, H., Gay, G., Davidson, B., & Ingraffea, A. (2002). Technology acceptance and social networking 

in distance learning. Educational Technology & Society, 6 (2), 50-61. 

Leicht, K. T. & Fennell, M. L. (2001). Professional work: A sociological approach, Malden, MA: Blackwell 

Publishers. 

Lewis, J., Marjoribanks, T. & Pirotta, M. (2003). Changing professions: General practitioners' perceptions of 

autonomy on the frontline. Journal of Sociology, 39 (1), 44-61. 

Lockwood, F. (2001). Innovation in distributed learning: creating the environment. In Lockwood, F. & Gooley, 

A. (Eds.) Innovation in open and distance learning: successful development of online and Web-based learning, 

London: Kogan Page, 1-14. 

Lockwood, F. & Gooley, A. (Eds.) (2001). Innovation in open and distance learning: successful development of 

online and Web-based learning, London: Kogan Page 

Mason, R. (1998). Globalising education, London & New York: Routledge. 

Mazmanian, P. E. & Davis, D. A. (2002). Continuing medical education and the physician as a learner. Journal 

of American Medical Association, 288 (9), 1057-1060. 

McConnell, D. (2000). Implementing computer supported cooperative learning, (2 nd ed.), London & Sterling: 

Kogan Page. 

Mick, D. G. & Fournier, S. (1998). Paradoxes of technology: consumer cognizance, emotions, and coping 

strategies. Journal of Consumer Research, 25, 123-143. 

61

Mitchell, E. & Sullivan, F. (2001). A descriptive feast but an evaluative famine: Systematic review of published 

articles on primary care computing during 1980-97. British Medical Journal, 322, 279-282. 

Mott, V. W. & Daley, B. J. (Eds.) (2000). Charting a course for continuing professional education: Reframing 

professional practice, 86, San Francisco, Jossey-Bass Publishers 

Patton, M. Q. (1990). Qualitative evaluation and research methods. Newbury Park, CA: Sage Publications. 

Peterson, M. W. (1999). Continuing medical education on the Internet: State of the art. Journal of Continuing 

Education in the Health Professions, 19, 242-249. 

Richards, B., Bolton, P., Veale B., & Quinlan, F. (1999). Information technology in general practice. Canberra: 

Commonwealth Department of Health and Aged Care. 

Spector, J. M., & Wang, X. (2002). Integrating technology into learning and working: Promising opportunities 

and problematic issues. Educational Technology & Society, 5 (1), 1-7. 

Strauss, A., & Corbin, J., (1998). Basics of qualitative research. Techniques and procedures for developing 

grounded theory. Thousand Oaks, CA: Sage Publications. 

Western, M., Dwan, K., Makkai, T., del Mar, C., & Western, J. (2001). Measuring IT Use in Australian General 

Practice 2001. General Practice Computer Group, Canberra: Department of Health and Ageing. 

Wickramasinghe, N. & Lamb, R. (2002). Managing managed care: The enabling role of IS/IT. Journal of 

Management in Medicine, 16 (2/3), 238-250. 

Wilson, A. L. (2000). Professional practice in the modern world. In Mott, V. W. & Daley, B. J. (Eds.) Charting a 

course for continuing Professional education: Reframing professional practice, 86, San Francisco: Jossey-Bass 

Publishers, 71-79. 

62

DiPietro, K. (2004). The Effects of a Constructivist Intervention on Pre-Service Teachers. Educational Technology & Society, 7 

(1), 63-77. 

Abstract 

The Effects of a Constructivist Intervention on Pre-Service Teachers 

Kathryn DiPietro 

Assistant Professor, Technology-Based Teacher Education 

Lehigh University, 111 Research Drive, Iacocca Hall Room A-111 

Bethlehem, PA 18015, USA 

Tel: +1-610-758-3236 

Fax: +1-610-758-3243 

kad9@lehigh.edu 

The purpose of this study was to determine the effect of pre-service teachers’ participation in a constructivist 

intervention supported by technology on their confidence in their own ability to plan and create six technologysupported, 

constructivist, learning activities, as well as to understand their perceptions of the experience. 

Participants were 23 pre-service teachers accepted into the College of Education’s Masters program at the 

University of Tennessee Knoxville and enrolled in an introduction to instructional computing course during the 

summer of 2001. 

A survey was used to assess pre-intervention confidence levels and experience with six technology-supported, 

constructivist, learning activities. Students were then situated in a class that employed constructivist 

methodology to facilitate their own exploration of constructivist pedagogy supported by technology. Once 

students completed the class, they were asked to re-take the survey. A paired samples t-test was used to compare 

pre-intervention confidence levels with post-intervention confidence levels. The results revealed a significant 

difference, p < .001, in each of the six areas. 

Journals, focus groups, and interviews were used to gain insight into the participants’ perceptions of the 

experience and suggested a reflective process. 

Keywords 

Technology, Teacher education, Educational technology, Constructivism 

Introduction 

As early as 1983, partially in response to A Nation at Risk (The National Commission on Excellence in Education, 

1983), educators began to look for ways to resolve lagging student performance, including the use of computers and 

technology. Efforts nationally to advocate and promote the dissemination and use of instructional technology have 

included Congress’ passage of the Technology for Education Act of 1994, which asserted technology’s value as a 

critical instructional tool and prompted the development and adoption of many national and state technology plans; 

and the creation of The Department of Educational Technology to oversee and guide the infusion of technology into 

educational systems. Since 1995, over $8 billion has been allocated in federal funds alone to assist in integrating 

computers into education. In one year, 1999-2000, educational technology in K-12 received over $5.67 billion in 

state and local funds. Due in part to these incentives, the percentage of schools with microcomputers has increased 

dramatically, from only 30% in 1982 to over 98% in 2000 (Market Data Retrieval, 2000). Over 94% of schools are 

connected to the Internet, and over 80% of teachers have Internet access in their classrooms (Market Data Retrieval, 

2000). Also, the student-to-computer ratio has decreased from 125-to-1 in 1981, to approximately 5-to-1 in 2000 

(Market Data Retrieval, 2000). “On the average schools offer 19 hours of technology-related professional 

development” per year (Market Data Retrieval, 2001). 

Problem 1: Inadequate technical preparation 

However, counting computers and Internet connections does little to address the more important question of how– 

and even whether–educators are using technology. In general, educators have not adopted technology as an 

63 





specific permission and/or a fee. Request permissions from the editors at kinshuk@massey.ac.nz. 

kinshuk@ieee.org.

instructional tool. At the Secretary’s Conference on Educational Technology, Goldman, Cole, and Syer (1999) noted 

while applications such as spreadsheets, simulations, CAD systems, or multimedia software are often used for 

limited tasks like word processing, they are rarely used for content learning. A 1998 report from the Office of 

Technology Assessment (OTA) found that fewer than half of all teachers were using computers in their teaching, 

despite beliefs about their effectiveness and desires to use them. 

A report by Scheffler & Logan (1999) suggests a reason for this gap between beliefs and practices: although most of 

the teachers surveyed expressed the belief that technology is a valuable and important teaching tool, fewer than 20% 

felt adequately prepared to integrate technology into the curriculum. Recent surveys confirm that many practicing 

teachers do not feel well prepared to integrate technology in instruction (Education Week, 1999; The National Center 

for Educational Statistics, 1998). 

One might hope that the problem of inadequate technical preparation would recede as new teachers enter the 

profession. According to estimates from the National Council for Accreditation of Teacher Education (NCATE), 

over two million teachers, comprising over half of the nation’s educators, will be hired over the next decade 

(Education Week, 1999; Gerald & Husser, 1991). NCATE currently requires all teacher education institutions to 

include technology training in their programs and further emphasizes technology as central to the teacher preparation 

process (Wise, 1997). By 1999, forty-two states required a course in instructional technology as part of their teacher 

education programs. Milkin (1999) reports that 70% of teacher training programs require students to take three or 

more credit hours of technology-focused courses. While these trends appear promising, a more recent survey by 

Milkin (2001) indicates that pre-service teachers, like their colleagues already in the classroom, are not adequately 

prepared to integrate technology into their teaching practices (Milken, 2001). 

Problem 2: Persistent inadequacy of technology training for new teachers 

What can explain the persistent inadequacy of technology training for new teachers? One problem may be that 

teachers have few good role models. Even when technology is readily available, faculty in teacher education 

programs often fail to use technology in their own research or teaching. A study by Persichitte, Tharp, and Caffarella 

(1997) found that only 45% of education program faculty use technologies in their own classes. Further, only 40% of 

students enrolled in teacher education programs are required to design and deliver instruction using technology. 

When asked the extent to which they exposed their pre-service teachers to technology in their classes, field 

experiences, and curriculum materials, the majority of faculty members at 416 teacher preparation institutions 

disclosed that they generally do not practice or model the use of technology (Milken, 2001). The problem is 

exacerbated by the fact that many colleges of education do not have technology-enhanced classrooms that would 

allow faculty to routinely model the use of the Internet and other technologies (Milken, 2001). This is a case of “do 

as I say, not as I do.” 

Problem 3: Content of technology courses 

Another problem is the content of the technology courses that schools of education mandate for their students. In a 

2000 survey, Hargrave and Hsu investigated instructional technology courses at 88 institutions of higher education 

belonging to the Holmes Group, a national consortium of research institutions “committed to making programs of 

teacher preparation more rigorous and connected to liberal arts education, research on learning and teaching, and 

wise practice in schools” (p. 305). Seventy-three percent of the colleges reported offering a specific introduction to 

instructional technology course. At 60% of these colleges, the course was three credit hours with three contact hours 

per week. Eighty-three percent of those colleges reported that the course was taught in a lecture and lab format, and 

no prerequisites were required for enrollment in the course. The primary focus of these courses was computer 

technology. While over 50% of the institutions reported addressing and using computer-based instruction (packaged 

software such as drill and practice, tutorials, educational games, problem-solving, and simulations), the majority of 

the institutions identified classroom design, needs analysis, audience analysis, task analysis, and situated cognition as 

topics not covered in the courses. Yet, teachers are expected to use technology to provide meaningful studentcentered 

curriculum-oriented learning activities for their students. The question of “how” begs an answer. 

64

Clearly, focusing on technology skills alone does little to move teachers to a point where they can use technology 

meaningfully in their classrooms. Traditionally, both in-service and pre-service technology training programs have 

focused on software instead of curriculum, leaving teachers unable to create or implement learning activities that use 

technology meaningfully (Gilmore, 1995; Moersche, 1995; Moursand & Bielefeldt, 1999; Yildirim, 2000). These 

skills-based and software-based approaches leave teachers without a clear vision of how technology can improve 

teaching and learning (Office of Educational Research and Improvement, 1993). Training that focuses on specific 

technologies or the mechanics of computer technology has little carry-over into classrooms (Beavers, 2001). 

Urgently needed is a change from a skills-based approach to an approach that incorporates technology seamlessly 

into subject matter in much the same way that the practical applications of technology have permeated society 

(Williams & Williams, 1997). Teachers should be introduced to a diverse range of technology and a variety of 

applications, with a focus on the development of creativity, adaptability and collaborative problem-solving skills 

(Williams & Williams, 1997). The integration of technology happens when tools are presented in the context of 

meaningful authentic learning situations, where users can see practical applications, engage in reflective teaching, 

and share their ideas with others (Spady, 1994; Warner, 1999). 

Implications for teaching technology 

What are the best practices for teaching technology? Research indicates that teachers whose pedagogical beliefs are 

consistent with constructivist learning theory are more likely to use technology in their practices. Fisher (1997) 

surveyed 287 Colorado public school teachers to determine the degree of importance they assigned to 10 technology 

literacy competencies. These teachers viewed the ability to use constructivist teaching pedagogy supported by 

technology as the most critical technology competency. Becker (1999) surveyed approximately 2,250 teachers to 

determine Internet use. As part of the study, Becker examined teachers’ pedagogical beliefs and practices in 

relationship to Internet use. What he found was “the more constructivist the teacher the greater their average use and 

the more positively they viewed the Internet” (para. 56). Teachers’ pedagogical beliefs and understanding of 

constructivistism are critical factors for determining if and how technology is used to enhance learning. 

Evidence supports an approach that is integrated and provides students not simply with a single instructional 

technology course, but also with methods and elective courses that integrate and model technology throughout the 

program (Todd, 1993; Wetzel, 1993). Institutions that provide routine opportunities for using technology in everyday 

classroom and practicum experiences report the highest level of student technology skills (Milken, 2001). Skills and 

processes are best learned when they are not taught in isolation, but are acquired within the context of accomplishing 

meaningful tasks (Harel & Papert, 1991). In identifying the single change that transformed San Carlos School in 

Monterey, California, Adams (2000) points to the application of technologies directly and relevantly to classrooms 

(job-embedded training): “Our technology center ceased to be viewed as a separate entity and has become an 

extension of each teachers’ classroom. What the technology is being used for comes directly from the classroom 

curriculum” (p. 116). Calls for job-embedded learning are echoed through the literature (Sparks & Hirsch, 1997). 

Drawing on adult learning theory, McKenzie (2001) emphasizes the importance of making the learning experience 

self-directed, contextual, and relevant both to personal interests/needs and to daily practice. Learning to use 

technology not only involves the acquisition of computer skills, but also is a process whereby students try, fail, 

access, evaluate, analyze and apply skills meaningfully (McKenzie, 2001; Scheffler & Logan, 1999). Pre-service 

teachers judge the potential or usefulness of computing by using a relevancy-irrelevancy dimension (Laffey & 

Musser, 1998). They want to know how to apply the technology skills in the context of learning and teaching. They 

tend to view technology as an add-on and rely on external factors and facilitators to infuse technology into their 

classrooms (Beyerbach, Walsh, & Vannatta, 2001). Technology, therefore, needs to become a personal tool for preservice 

teachers, one they use expressively, creatively, meaningfully and purposefully. Teachers who personally use 

and own computers are likely to feel less anxious about integrating them into their practices (Laffey & Musser, 1998; 

Hochman, Maurer, & Roebuck, 1993; Kearns, 1992). 

Because there are various definitions of the term constructivism, it is important for the researcher to provide a 

working definition. A constructivist approach “involves having students work on complex projects, often in groups, 

and often with different groups working on different projects. In this model, students learn skills and concepts in the 

context of using them to do something—for example, in making a project. These projects follow from a 

65

constructivist theory of learning that suggests that subject matter becomes meaningful, and therefore understandable, 

only when it is used in context-rich activities. Teachers whose instructional plan follows from constructivist learning 

theory not only use group projects more than other teachers; they will, for example, emphasize the student’s own 

responsibility for designing their own tasks, for figuring out their own methods of solving problems, and for 

assessing their own work —all as a means of making learning tasks more meaningful to students” (Becker, 1999, 

para. 53). Learning, as such, moves beyond skills that are externally imposed and irrelevant to the learners’ lives, to 

contextual, personally relevant processes of exploration, manipulation of information, and investigation of 

possibilities (Robyler & Edwards, 2000; Wilson, Teslow, & Osman-Jouchoux, 1999; Jonassen, 1994). Under these 

assumptions, meaningful learning occurs when the goal of education is to “help students learn how to recognize and 

solve problems, comprehend new phenomena, construct mental models of those phenomena, and given a new 

situation, set goals and regulate their own learning” (Jonassen, Peck, & Wilson, 1999, p. 3). Technology, when used 

in a constructivist manner, becomes a tool that students learn with. Learners use technologies to manipulate data, to 

explore relationships, to intentionally and actively process information, to construct personal and socially sha 

red meaning, and to reflect on the learning process (Jonassen, Peck, & Wilson, 1999). 

Purpose of the study 

The purpose of this study is to determine the effects of situating pre-service teachers in a constructivist learning 

environment supported by technology. From the problems discussed above, the following research questions emerge: 

What is the effect of an instructional approach that focuses on constructivist pedagogy supported by technology 

on pre-service teachers’ confidence in their own ability to plan and develop constructivist learning activities 

supported by technology? 

How do pre-service teachers perceive the experience of participating in an instructional technology course that 

employs a constructivist pedagogical approach and uses? 

Methods and Procedures 

Constructivism and adult learning theory provided the conceptual framework for the study and guided the design, 

execution, and framework for the analysis of data. Using a quasi-experimental approach, this study explores the 

effect of a constructivist pedagogical intervention on the confidence of pre-service teachers in their own ability to 

develop constructivist learning activities that use technology to enhance learning as well as the participants’ 

perceptions of the experience. 

The participants in the study were 23 pre-service teachers at the University of Tennessee Knoxville (UTK) College 

of Education’s teacher education program who were enrolled in a Summer 2001 course titled Instructional 

Technology, Curriculum, and Evaluation 486: Introduction to Instructional Computing (ITCE 486). To answer the 

research questions, a variety of methods were used: pre- and post-intervention surveys, journal entries, interviews, 

and focus groups. Participants were given a pre-intervention survey and asked to rank their confidence in their ability 

to plan and create various constructivist learning activities supported by technology. 

Drawing on constructivism and adult learning theory, the intervention was designed to situate participants in a course 

that would both model and teach constructivist methods as well as how those methods could be supported by 

technology. These learning activities were developed with five attributes of constructivist learning in mind: they 

were active, constructive, intentional, authentic, and cooperative (Jonassen, Peck, and Wilson, 1999). Each of these 

attributes were defined: 

Active (Manipulation/Observant): Participants “actively [manipulate] the objects and tools of the trade and 

[observe] the effects of what they have done” (p. 8). 

Constructive (Articulative/Reflective): Participants construct meaning by reflecting on the process and 

articulating their experiences and conceptual understandings. 

Intentional (Reflective/Regulatory): Participants engaging in intentional learning while trying to achieve a 

cognitive goal, reflecting, evaluating, and articulating the process, “decisions they make, strategies they use, and 

the answers they found” (Jonassen, Peck, & Wilson, 1999, p. 9). 

Authentic (Complex/Contextual): Participants engage in learning activities that are complex and contextual. 

66

Cooperative (Collaborative/Conversational): Participants engage in collaborative activities during which they 

dialog about a task, the methods they will use to accomplish the task, as well as seeking out alternative ideas and 

opinions. 

An example of a learning activity that met this criteria was to ask elementary science pre-service teachers to become 

familiar with insects that are native to East Tennessee by spending twenty minutes outside observing insects, taking 

digital pictures of the insects, documenting the behavior of the insects, and researching the insects online or using 

field guides. From these data, elementary science pre-service teachers were asked to draw conclusions about insects 

in East Tennessee and then sharing their findings with the class via a slideshow presentation (see Appendix A). 

Similarly, secondary art pre-service teacher were ask to investigate the relationship between form and function in 

architecture by exploring campus buildings with unique designs, documenting those designs with a digital video 

camera, and finding out the use of the building by walking around in the building and taking field notes. Students 

were then asked to draw conclusions about form and function of building and asked to create a slideshow 

presentation using the pictures they had taken to support their conclusions (Appendix B). It was from this framework 

that students both explored learning strategies and the use of technology to support learning. It was during activities 

such as these that students acquired technology skills—within the context of using them. The emphasis of these 

activities was always on exploring content or concepts. For each of the technologies (web-based inquiry, slideshows, 

database, and spreadsheets) similar learning activities were developed by the researcher for each grade level and 

subject area of the pre-service teachers. These are identified as integrated constructivist learning activities aligned to 

curriculum standards supported by technology. 

Once students completed these learning activities in teams, individually they developed their own learning activities 

that were aligned to curriculum standards, fit within an interdisciplinary unit theme, that were designed to engage 

learners in similar constructivist learning activities. Through their work on a series of activities, students produced an 

interdisciplinary thematic curriculum-based instructional unit that contained a series of constructivist lessons 

supported by those technologies (listed above). 

After participating in this course, (see Table 1 note for activities), they were given a post-intervention survey and 

again asked to rank themselves in the same areas, in order to determine the effect of the intervention. The surveys 

used a five-point Likert Scale, and were developed by the researcher using technology standards set by the 

International Society for Technology in Education’s (ISTE) National Educational Standards (NETS): Professional 

Preparation Profile (ISTE, 2000) as well as course goals developed collaboratively during team meetings Fall 

semester 2000 and Spring semester 2001 by instructors of the course. A two-sample paired t-test was used to analyze 

the difference in pre-intervention and post-intervention results. Because one of the goals of this study was to give a 

voice to pre-service teachers situated in a course that used constructivist methodology as well as to understand their 

perceptions of being situated in such a course, participants were asked to keep a weekly journal of their thoughts 

about the experience. Once data from the journals were gathered and analyzed, themes were identified and shared in 

focus groups with all the members of the study or used to develop interview questions. Journal entries, focus groups, 

and interview data were used to reveal participants’ experience of being situated in the course. 

Findings 

The data gathered via a pre- and post-intervention survey revealed a significant difference in participants’ pre- and 

post-intervention confidence in their abilities to plan and create various constructivist learning activities supported by 

technology, p < .001 (Table 1: Paired-Samples T-Test for Pre- and Post Survey). The Cronbach alpha reliability of 

this instrument is .93 and was calculated using the six items on the pre-test that related to confidence in ability to 

plan and create constructivist learning activities supported by technology. 

The following is an explanation of pre- and post-survey items: 

Pair 1, Pst 1, and Pre 1: Plan and create a multidisciplinary unit with constructivist learning activities supported 

by technology; 

Pair 2, Pst 2, and Pre 2: Plan and create a constructivist learning activity supported by a slideshow; 

Pair 3, Pst 3, and Pre 3: Plan and create an inquiry-based learning activity supported by a WebQuest; 

67

Pair 4, Pst 4, and Pre 4: Plan and create a constructivist learning activity that makes use of categorizing, sorting, 

and classifying supported by a database; 

Pair 5, Pst 5, and Pre 5: Plan and create a constructivist learning activity that makes use of predicting, 

hypothesizing, and calculating supported by a spreadsheet; 

Pair 6, Pst 6, and Pre 6: Identify and evaluate resources for constructivist learning activities. 

Table 1: Paired-Samples T-Test for Pre- and Post Survey 

Source DF m SD T P 

Pair 1 14 1.53 .99 5.996

I found it challenging to create a way for students to use PowerPoint since it is unreasonable for 

them to actually use the computers. (A few may have that capability, but most will not have the 

coordination or skill level to use keyboards, etc.) However, I did come up with a project- modified 

from a paper project- that allows students to participate and use PowerPoint. 

Participants thought about their experiences, worked through the process of adapting their ideas about teaching and 

learning and evaluated how their new experiences and ideas might work in their future classrooms and how they 

might affect their future students. A pre-service science teacher analyzed how a constructivist activity supported by a 

slideshow might work in a future class: 

I am very excited about all of the technology and skill I am learning to bring to the classroom. My 

concern is will we have enough computers and technology to make this a viable learning process 

for the classroom? I would like to hear more about how we are going to accomplish these activities 

if we have limited computer time and availability. I’m thinking we may need to rotate projects and 

computer use. For example, in the science classroom, I might have students working on different 

projects at the same time. Some might be involved in hands-on inquiry while others are preparing 

their PowerPoint presentations. This would make sense, except it might be hard to keep everyone 

on track and be available to assist the students as they need help. 

Throughout the intervention, participants engaged in “group thinking,” self- reflection about and evaluation of their 

feelings when situated in cooperative groups in terms of their relationships with others and in terms of the 

application to their learning. In general, they were positive about working in a collaborative setting in terms of 

relationships with others, although there were times when students felt awkward or that the work in the group was 

not equally distributed. Participants valued getting to know each other and being members of a group, and having 

colleagues with whom to share ideas. This sense of community facilitated the sharing of ideas both within and 

among groups. The group process was seen as a way to capitalize on the strengths of each person. Working in teams 

improved participants’ confidence in their ability to complete learning activities. One student with limited 

technology experience described his feelings when he completed a group assignment: 

This was most productive for me because I got through this PowerPoint presentation, and in my 

opinion, did quite well. I am not a huge fan of computers, they intimidate me, but I was able to 

conquer this challenging task. I got finished with my presentation on Wednesday and it is due on 

Monday, so I feel very accomplished. My self-esteem benefits when I can do something that I 

previously thought would be nearly impossible for me. I appreciate the opportunity to work in 

groups on the PowerPoint (the insect assignment). That was a much better way to lead us into use 

of this technology than simply handing out a tasksheet, or instructions, and leaving us to fend for 

ourselves. I knew the value of the group work, so I tried to absorb all that I could about using 

PowerPoint by watching (names group member) who understood it better than me.... The 

atmosphere is supportive not scary. We are not a class full of students all worried about our grades 

and being nervous about our abilities not being good enough to get us the grade we want. No, we 

are more relaxed (which in my opinion makes for a better learning environment. 

Participants engaged in metacognition by spending time reflecting on their learning processes (“process thinking”) in 

a constructivist learning environment. They generally agreed that the constructivist learning environment challenged 

them to be self-reliant in evaluating and adjusting their learning and learning strategies. Participants evaluated their 

knowledge and skills and identified and developed processes by which to acquire new knowledge and skills. Further, 

they evaluated those processes and their new knowledge and skills. Participants felt the focus of the course was on 

processes. Because they felt that the focus of their efforts should be processes, self-directed learning, and selfevaluation 

rather than products, they were motivated to do more than the minimum to complete an assignment. A 

special education major who worked for several years as a technology lab aide in an elementary public schools 

noted: 

It is like an artist’s canvas. Okay, so you know what you have to come up with so now you have to 

actually come up with it. And people who don’t have the opportunity to do it that way get stuck in 

a box and all they do is fill in templates and fill out forms and they don’t see the ability to go 

beyond and create beyond what’s there. Because they don’t know how to get there 

69

A process-related theme that reoccurred throughout the journals and interviews was creativity. Students felt that 

constructivist methodology allowed and encouraged them to be more imaginative and creative. As one participant 

said, “It’s like giving you the guidelines, being there to help you if you have difficulties. But it was a lot of selfmotivation. 

It’s a method that is going to leave us a lot more flexible and a lot more creative.” Another pre-service 

science teacher discussed how constructivist methods influence creativity, 

We are all going to be different, we’re all going to have different styles, we are all going to enjoy 

doing different things and out students are going to enjoy doing different things. So it’s a method 

that is going to leave us more flexibility and a lot more creativity and a lot more room to use things 

in the way that is going to suit our style. Whereas you know, the directive method is like you need 

to stamp out like a cookie-cutter. 

Participants spent time thinking about the instruction (“instruction thinking”). They evaluated the instructor and her 

role in their learning. Students valued the freedom that the instructional approach gave them to chart their own 

learning and acquisition of skills. They felt the method allowed them to explore topics at a greater depth, but wanted 

to know that the instructor or peers would support them if needed. One student wrote: “You are trying to let us figure 

out what to do after a little guidance from you. I appreciate that a lot and think that is a great way of teaching. Do it 

for them and they will never learn. Show them and they will never learn. Let them do it hands on, they will learn.” 

They valued the collegial approach, noting with approval that their own ideas were as valued as the instructor’s, they 

were viewed as peers, and they were encouraged to build active partnerships in learning, sharing their knowledge and 

teaching each other. One student described this: 

She’s very open to, ‘No, I don’t know everything, I don’t know every version of software, I’m not 

supposed to. That’s what help is for. You know, let’s create, let’s do this, let’s go in and figure out 

how to do it.’ A lot of teachers are so scared that they don’t know everything. They are afraid to let 

the kids on there because they don’t want to be in a position to say I don’t know. And that’s a 

really scary position to be in if you look at a little kid who thinks you are this really awesome 

person and say I don’t know, but there are lots of ways to say I don’t know, well I am not sure, 

let’s figure this out. I mean there is just so much to learn from each other. When one of us comes 

up with something that she does not know, oh that’s cool, I hadn’t seen that before! and really 

encourages us to explore and discover and go beyond what she’s shown us. 

Participants also wrote of feeling supported or valued by the instructor and their peers. They appreciated having 

access to the instructor via e-mail as well as via phone and office hours, receiving prompt responses to their 

concerns, questions, and thoughts even outside class. Although they generally felt the instructor was accessible to 

them in class, they did voice some frustration at having to wait for the instructor’s help during class. They valued 

being able to share ideas and discuss ideas with peers. One participant reflects on the value of sharing with peers: “It 

is a really good environment. There seems to be a lot of peer support also. The young man who sits next to me, 

(name), is so very helpful. There seems to be that cooperative spirit in this classroom and that is encouraging.” 

Students evaluated the materials used in the course and how they could adapt those materials for their own use both 

during the course and after the course. The website proved challenging initially because of an unfamiliar interface, 

but once students became acclimated to it, it provided them with a central location to access information about the 

course, assignments, and grades, and alerted them to daily agendas. They supplied on-going feedback about the 

organization of the website and appreciated seeing their suggestions incorporated into its structure. It was important 

that the website be kept current. In addition, they spent time analyzing the other materials used in the class -- the 

course packet, the book, instructor-created handouts, and software demos (free 30 day trial versions provided by 

publishers). They felt that the course packet was unnecessary. The course packet contained technology related 

articles. While they believed the book had useful information, they felt its usefulness was not commensurate with its 

expense. They suggested alternate resources that were not as costly, such as websites and handouts. 

Participants valued the instructor-made handouts that were specific to each assignment. Students commented on the 

usefulness of the skill tasksheets (addressing technology specifics as needed) that were created for each of the 

technology components. As one student put it, “She has a handout for every skill, every piece of software so that you 

could always kind of teach yourself, walk yourself through, and learn the software.” While having access to such 

reference materials was important, students did not want to be made to work through tasksheets, but to use them as 

70

needed in the content of creating projects or practicing skills. Participants commented on how they might use skill 

sheets with their future students or personally as they apply the skills they had acquired in their future teaching. 

Software demos were important to students because they felt the demos allowed and encouraged them to explore 

software outside of class. In some instances, they were frustrated because some of the software demos were not fully 

functional and because they encountered some problems with the platforms they were using, but the general 

consensus was that demo software was valuable and important to their developing their projects. 

Participants spent time thinking about the accessibility of technology, their own skill levels using technology, and 

their hardware and software successes and problems (“technology thinking”). They also compared their new 

experiences with their prior knowledge and experiences. While a list of critical technology skills was identified for 

students in the course packet, students did not use the list to assess their technology skills. Instead they assessed their 

skills contextually within the scope of the tasks they were accomplishing. They identified their deficit technology 

skills and how those skills affected their use of the technology to accomplish tasks. One student assessed his own 

Internet searching skills and how his lack of Internet searching strategies affected his ability to find information 

related to his unit theme of geology: 

One problem I am having using the Internet as a resource is that I end up finding lots of advertising 

material and less raw information. I need to know how to get sites that simply have lots of 

information on the subject of geology (like an online encyclopedia resource). I don’t want to end 

up using mediocre Internet resources because I don’t know how to best search the web. 

Students considered it important to have hands-on access to the technology they were learning. After participating in 

Activity 2: Constructivist Slideshow: Slopes which required learners to use a digital camera to document slopes in 

their environment, one student commented on how he felt when he was given a digital camera to use during the 

activity: 

I also had a chance Wednesday to use a digital camera. I have always wanted to use one but no one 

has ever let me. It may have not meant much to you to let me use it but it meant a lot to me. Using 

it was so awesome and it is so neat to know you can now keep pictures you want and throw away 

the ones you don’t before even developing them. 

Participants gauged their technology proficiency by comparison to their peers as well as to their own prior 

experiences. Although some of the students were proficient and experienced in using some of the technologies, they 

felt the constructivist environment allowed them to explore the technologies in greater depth. A participant with high 

technology skills acquired during her career in business, compared her skills to those of her classmates and also 

discussed challenging herself to learn an unfamiliar platform, 

This week was exciting and frustrating. As I talk with some of the other students, I find that I am 

much slower in catching on to the technology. Some of the others are commenting on the slow 

pace of the class. I, however, do not find the class to be moving too slowly. It seems about right for 

me, so I hope the pace does not pick up. I think one other difference exists between me and the 

others too. Most of the other students selected a computer that they are already comfortable with 

and are working on what they usually use at home. I decided to work with a Mac, because I know a 

lot of schools have Macs and I want to be able to use one if need be. That may also account for my 

slow pace. The Mac is not all that different, but it does take a little more thought. (journal entry 

from week two of the intervention). 

Having a choice of platforms was important to students. Generally students elected to use the platform that they were 

already most comfortable with or one that was compatible with the system they had access to outside of the class. 

Although students were free to sit wherever they wanted and use whichever platform they chose, three of the 

students in the class self-elected to use a platform that was unfamiliar to them because they saw value in learning 

both Windows and Macintosh operating systems. 

Participants discussed their access to computers, software, and other technologies. While noting minor problems, 

students felt technologies were generally accessible to them both in and out of the classroom. They noted that the 

computer lab hours, software, and hardware in the College of Education and other labs around campus met their 

needs. However, there was concern about the lack of access to color printers and other peripheral devices such a 

71

digital cameras and scanners. Students wanted to be able to use these technologies to complete their projects outside 

of class as well as having access to them in class. 

When students encountered hardware or software problems, they tried to work through the problems themselves. 

Participants described this process as playing around or exploring the software and felt the environment in the 

classroom allowed them to work in this way: 

I have realized my biggest problem is patience. I don’t like “playing” around on the computer. But 

I have found that this is a good way to learn my way around. I am a very procedural type person 

and I like to do things right the first time. This is an excellent tool, though, because my students 

will not do things right the first time, and if I can’t have the patience with myself, there is NO 

WAY I will have the patience with my students that I definitely must have. (I guess I’ll be doing 

some “growing” in this class, won’t I?!) 

If they could not solve their own problems, they relied on peers or the instructor, but generally only as a last resort. 

In sum, participants felt the intervention engaged them in a reflective process that led them to challenge their 

constructs of teaching. 

Conclusions 

A review of the literature indicates that a skills-based approach to teaching technology and its integration into the 

classroom has not had the desired outcome: it has not adequately prepared teachers to integrate computer technology 

into instruction as a tool for data analysis, exploration, simulation, or other high order thinking-skills. Learning is a 

complex process. Cognitive psychology reveals that learning is grounded in active participation, that knowledge is 

constructed, and that the situation and context influence not only what we learn, but also how we can use what we 

learn. To short cut the learning process by offering learners bits of decontextualized information divorces learning 

from reality. Direct instruction may not allow learners to go through the process of actively and intentionally 

constructing, adapting, and adjusting mental models. 

Situating pre-service teachers in an intervention that employs a constructivist methodology honors learning processes 

in several ways. First, it honors learning as a natural, adaptive process that allows students to act on and manipulate 

their environment and observe the results of their actions. (Jonassen, Peck, & Wilson, 1999). Many of the students in 

the study had either limited or no exposure to constructivist experiences or to planning and creating constructivist 

learning activities supported by technology. Thus their mental models of both were limited by their prior 

experiences. As evidenced in many of their journal entries, the students initially tried to fit their new experiences into 

their old teaching/learning paradigm. At times they admittedly clung to their pre-intervention mental models: “I 

found it challenging to create a way for students to use PowerPoint since it is unreasonable for them to actually use 

the computer.” The difference between their intervention experiences and their prior experiences and mental models 

of teaching, learning, and the use of technology created cognitive dissonance that was evident from their initial 

reports of confusion and frustration, of being intimidated and overwhelmed. These initial responses required learners 

to reflect upon, manipulate, self-regulate, and adjust their construct of teaching, learning, and the role of technology 

in their future teaching practices. 

For these pre-service teachers, being situated in a constructivist intervention evoked a reflective process: students 

analyzed their ideas about teaching and learning by comparing their new experiences to what they had viewed as 

effective teaching and learning before the intervention. They examined their own assumptions and tried to judge how 

those assumptions fit into their construct of teaching and learning. They actively and collaboratively adjusted their 

constructs of teaching and learning, thereby creating new mental models. They tested their assumptions against their 

constructs of how they would teach, how their students would learn, and how their adjusted models might work in 

their future classrooms. As students actively worked through this process, their confidence significantly increased. 

They became self-directed, self-regulated, and motivated. They felt valued and important. And most importantly, 

they were encouraged to be creative and imaginative, to explore, discover, and “go beyond.” In short, they wanted to 

learn. 

72

The question remains, “How can we best train teachers to use technology to facilitate student learning?” In this 

study, related literature has been reviewed; pre-service teachers have been asked to describe their own perceptions of 

an introduction to instructional technology class that used a constructivist approach; and changes in the confidence 

levels of pre-service teachers in their ability to plan and create constructivist activities supported by technology have 

been observed. The relationship between the increase in confidence among pre-service teachers and the literature 

related to pre-service training and technology and teacher beliefs, practices, and technology use is clear. Teaching 

technology skills in isolation has not been effective in training teachers to use technology to support learning, nor has 

direct traditional instruction in technology integration. However, when teachers are situated in a learning 

environment where constructivist instructional methods are modeled and implemented, they develop confidence in 

their abilities to produce constructivist learning activities that are tied to curriculum and supported by technology. 

Both this study and others, (Milken, 2001; Todd, 1993; Wetzel, 1993) suggest that teacher education programs need 

routinely to model the use of technology as an integrated teaching tool. That cannot happen when the technology is 

isolated in computer labs. It may happen when computers, software, and materials are integrated, physically as well 

as in their use, into methods classes or other classes where pre-service teachers learn. By using computers as a vital 

tool to facilitate their own learning, using computers to express themselves creatively, and using computers for 

personal purposes, teachers are more likely to learn to like using computers and find value in technology as an 

instructional tool (Laffey & Musser, 1998; Hochman, Maurer, & Roebuck, 1993; Kearns, 1992). People want to see 

relevance in what they are learning. For pre-service teachers, relevance means not only how they are personally 

affected, but also how methods and tools might affect the learning of their future students (Valdez, et al, 1999, 

Rodriguez & McDonald, 2001). Engaging pre-service teachers in the constructive processes of analyzing, adapting, 

testing, negotiating, retrying, and reflecting allows them to examine teaching and learning and to determine for 

themselves how to teach, how students learn, and how technology can support learning. 

References 

Adams, E. (2000). Transparent training and technology intuition. T.H.E. Journal, April, 115- 117. 

Beavers, D. (2001). Professional development outside the workshop box. Principal Leadership, May/June, 43-46. 

Becker, H. & Ravitz, J. (1999). The influence of computer and Internet use on teachers’ pedagogical practices and 

perceptions. Journal of Research on Computing in Education, 31(4). 356-384. 

Becker, H.J. (1999). Internet Use by Teachers. Retrieved April 24, 2001, from 

http://www.crito.uci.edu/tlc/findings/internet-Use/report.htm. 

Beyerbach, B., Walsh, C. & Vannatta, R. (2001). From teaching technology to using technology to enhance student 

learning: Preservice teachers’ changing perceptions of technology infusion. Journal of Technology and Teacher 

Education, 9(1), 105- 127. 

Collins, A., Brown, J., & Newman S. (1989). Cognitive apprenticeship: teaching the crafts of reading, writing, and 

mathematics. In Knowing, Learning, and Instruction: Essays in honor of Robert Glaser (pp. 453-494). Hillsdale, NJ: 

Lawrence Erlbaum Associates. 

Education Week (1999). Technology Counts ’99. Retrieved April 24, 1999, from 

http://www.edweek.org/sreports/tc99/. 

Eachus, P. & Cassidy, S. (1999). Developing the computer self-efficacy (CSE) scale: Investigating the relationships 

between CSE, gender, and experience with computers. Retrieved April 24, 1999, from 

http://www.salford.ac.uk/healthSci/selfeff/selfeff.htm. 

Fisher, M.M. (1997). The voice of experience: Inservice teacher technology competency recommendations for 

preservice teacher preparation programs. Journal of Technology and Teacher Education, 5, 139-147. 

73

Gerald, D. & Hussar, W. (1991). Projections of Educational Statistics to 2002. Washington, DC: National Center for 

Educational Statistics, U.S. Department of Education. 

Gilmore, A.M. (1995). Turning teachers on to computers: Evaluation of a teacher development program. Journal of 

Research on Computing in Education, 27, 251-269. 

Goldman, S., Cole, K., & Syer, C. (1999). The Technology/Content Dilemma. Retrieved May 16, 2001, from 

http://www.ed.gov/TechConf/1999/whitepapers/paper4.html. 

Harel, I. & Papert, S. (1991). Software as a learning environment. In I. Harel and S. Papert (Eds.) Constructivism, 

(pp. 41-84). Norwoord, NJ: Ablex. 

Hargrave, C. & His, Ying-Shao (2000). Survey of instructional technology courses for preservice teachers, Journal 

of Technology and Teacher Education, 8(4), 303-314. 

Hochman, A., Maurer, M., & Roebuck, D. (1993). Buttons and cards and fields, oh my! Tech Trends, 38(2), 25-28. 

Honeyman, D. & White, W. (1987). Computer anxiety in educators learning to use the computer: A preliminary 

report. Journal of Research on Computing in Education, 20(2), 129-138. 

International Society for Technology in Education (2000). National Educational Technology Standards for Students: 

Connecting Curriculum and Technology. Eugene, OR: International Society for Technology in Education. 

International Society for Technology in Education (2000). National Educational Standards (NETS) for Teachers 

Project. Retrieved June 29, 2001, from http://cnets.iste.org/intro_splash3.htm. 

Jonassen, D. (1994). Technology as Cognitive Tools: Learners as Designer. Retrieved August 13, 2001, from 

http://it.coe.uga.edu/itforum/paper1/paper1.html 

Jonassen, D. (1994). Thinking technology: Toward a constructivist model. Educational Technology, 34(4), 45-53. 

Jonassen, D., Peck, K., & Wilson, B. (1999). Learning with Technology: A constructivist perspective. Upper Saddle 

River, NJ: Prentice-Hall, Inc. 

Kearns, J. (1992). Does computer coursework transfer into teaching practice? Journal of Computing in Teacher 

Education, 8(4), 29-34. 

Koohang, A. A. (1987). A study of the attitudes of preservice teachers toward the use of computers. Educational 

Communication and Technology Journal, 35(3),145-149. 

Laffey, K. & Musser, D. (1998). Attitudes of preservice teachers about using technology in teaching. Journal of 

Technology and Teacher Education, 6(4), 223-241. 

Liu, M., Reed, W. & Phillips, P. (1992). Teacher education, students, and computers: Gender, major, prior computer 

experience, occurrence, and anxiety. Journal of Research on Computing in Education, 24(4), 457-467. 

Market Data Retrieval (2001). K-12 Schools Make Major Changes in Technology. Retrieved May 29, 2002, from 

http://www.schooldata.com/pr22.html. 

McKenzie, J. (2001). How Teachers Learn Technology Best. Retrieved September 8, 2002, from 

http://fno.org/mar01/howlearn.html. 

Milken Family Foundation (2002). Information Technology Underused in Teacher Education. Retrieved September 

21, 2003, from http://www.mff.org/edtech/article.taf?_function=detail&Content_vid1=131. 

74

Office of Educational Research and Improvement (1993). Using technology to support educational reform. 

Washington, DC: U.S. Government Printing Office. 

Moersch, C. (1995). Levels of technology implementation (LoTi): A framework for measuring classroom technology 

use. Learning and Leading with Technology, 23(3), 40-41. 

Moursand, D. & Bielefeldt, T. (1999). Information Technology in Teacher Education. Eugene, OR: International 

Society for Technology in Education. 

National Center for Educational Statistics. (1993). Edsearch: Educational statistics on disk. [Computer Software]. 

U.S. Department of Education. 

Persichitte, K., Tharp, D. & Caffarella, E. (1997). The Use of Technology by Schools, Colleges, and Departments of 

Education 1996. Washington, DC: American Association of Colleges for Teacher Education. 

Roblyer, M, & Edwards, J. (2000). Integrating Educational Technology into Teaching. Upper Saddle River, NJ: 

Prentice Hall. 

Rodriguez, S. & McDonald, J. (2001). Implementing a web-based middle school science curriculum: An Evaluation 

Report. Paper presented at SITE 2001, Orlando, FL. 

Scheffler, F. & Logan, J. (1999). Computer technology in schools: What teachers should know and be able to do. 

[Electronic version}. Journal of Research on Computing in Education, 31, 305-325. 

Spady, w. (1994). Choosing outcomes of significance. Educational Leadership, 51(6), 18-22. 

Sparks, D. & Hirsh, S. (1997). A New Vision for Staff Development. Oxford, OH: National Staff Development 

Council. 

Todd, N. (1993). A curriculum model for integrating technology in teacher education courses. Journal of Computing 

in Teacher Education, 9(3), 5-11. 

Valdez, G., McNabb, M.., Foertsche, M., Anderson. M., Hawkes, M., & Raack, L. (1999). Computer based 

technology and learning. Retrieved May 12, 2001, from http://www.ncrel.org/tplan/cbtl/toc.htm. 

Warner, M. (1999). Training today’s teachers for tomorrow’s classrooms. T.H.E. Journal, October. 

Wetzel, K. (1993). Models for achieving computer competencies in preservice education. Journal of Computing in 

Teacher Education, 9(4), 4-6. 

Wise, A. (1997). A message to NCATE institutions, board members, constituentorganizations, and friends about 

Technology and the New Professional Teacher: Preparing for 21 st Century Classrooms. Washington, DC: NCATE. 

Williams, A. & Williams, P. J. (1997). Problem-based learning: An appropriate method for technology education. 

Research in Science & Technological Education, 15(1). 

Wilson, B., Teslow, J., & Osman-Jouchoux R. (1995). The impact of constructivism (and postmodernism) on 

instructional design fundamentals. In B. Seels (Ed.) Instructional Design Fundamentals: A review and 

reconsideration (pp. 137-157). Englewood Cliffs, NJ: Educational Technology Publications. 

Woodrow, J. E. (1990). Locus of control and student teacher computer attitudes. Computers in Education, 14(5), 

421-432. 

Yildrim, S. (2000). Effects of an educational computing course on pre-service and inservice teachers: A discussion 

and analysis of attitudes and use. Journal of Research on Computing in Education, 32, 479-497. 

75

Appendix A 

Bugs 

Group Members: 

Context: 

Your class is in the middle of a unit on insects. Your teacher wants you to become familiar with the insects that are 

native to East Tennessee. Once you become a "bug expert," you will share your findings with the class by creating a 

book using a slideshow. 

Task: 

When you are done with this activity, you will: 

1. Have an idea of some of the insects that are native to East Tennessee. 

2. Have created a slideshow book that documents your findings and provides information about each of the insects 

you have identified. 

Process: 

• Get in your group and read the poem "Neighbors" by Marchette Chute. In that poem Marchette Chute lists 

several types of insects that she saw while watching. 

• Make a list of the type of insects you think you might see outside the classroom in East Tennessee. 

• Take the digital camera and some paper, go outside, find a nice quiet place in the grass, and spend about 20 

minutes observing a patch of grass. Take pictures of any insects you see. Make field notes about where and 

when you saw the insect and its behavior. After about 20 minutes, come back in the classroom. 

• Sort your pictures. Use field guides or the Internet to try to identify each insect that you observed. Draw 

some conclusions about insects in East Tennessee based on your observations. 

• Use a slideshow to create a book about the insects you observed. Make an introduction page and a page for 

each insect that you observed. On the page about each insect include your picture of the insect, your field 

notes, and any other information that you found about the insects that you'd like to include. If you use any 

outside sources, be sure to cite them (tell the source). 

Evaluation: 

You will be evaluated according to the following: 

• Did all members of your team participate and work as a team? 

• Did you spend 20 minutes outside observing an area for insects? Did you take field notes about the insects you 

observed? Did you take pictures of the insects you observed? 

• Did you identify the insects you observed or make a reasonable effort to identify them? 

• Did you create a book that includes a title page and a page for each insect with your field notes, the photographs, 

and any other information you found about the insects? 

• What conclusions can you draw about insects in East Tennessee? 

• Describe the process your group used to complete the activity. 

76

Appendix B 

Forms and Function in Architecture 

Group Members: 

Context 

You are in a high school art class. Your class is in the middle of a unit on architecture. Your teacher has asked you to 

look at forms that are found in buildings and how those forms serve a purpose. 

Task: 

When you have finished this assignment, you will have: 

• Investigated the relationship between form and function in architecture. 

• Create a slideshow presentation that illustrates the relationship between form and function in building on the 

UTK campus. 

Process: 

• In your group gather the following resources: 

• A digital camera with a few disks 

• Paper and Pencil 

• Now go outside and walk around the campus take 4 pictures of 4 different buildings. Select buildings that have 

unique designs. Now find out what the building is used for and what happens inside that building. You may need 

to walk inside the buildings to investigate. Make notes about the buildings as you are taking 

• Come back to the classroom and in your group, discuss the form of the buildings and they way the building is 

used. Are there any conclusions that you can draw about the way the architect designed the building and its use? 

Do you think the architect used had the purpose of the building in mind when he designed it? What makes you 

think he/she did or did not? 

• Use a slideshow to create a presentation that uses your pictures to illustrate your conclusions about the 

relationship between the form and the function of the buildings. Be sure to have a title slide and then content 

slides for each building. Add any other slides that you feel are necessary. 

Evaluation: 

You will be evaluated according to the following criteria: 

• Did your team work as a group and each member participate? 

• Did you spend about 30 minutes walking around campus taking pictures of at least 4 buildings and investigate 

the purpose or use of the building? 

• Did your group create a slideshow presentation that includes a title page and content pages that uses your 

pictures to illustrate your conclusions about the relationship between the form and the function of the buildings? 

• Describe the process your group used to complete the activity. 

77

Myers, C. B., Bennett, D., Brown, G. & Henderson, T. (2004). Emerging Online Learning Environments and Student 

Learning: An Analysis of Faculty Perceptions. Educational Technology & Society, 7 (1), 78-86. 

Emerging Online Learning Environments and Student Learning: 

An Analysis of Faculty Perceptions 

Carrie B. Myers 

Department of Education, Montana State University 

Bozeman, MT 59717-0288 USA 

cbmyers@montana.edu 

Dennis Bennett, Gary Brown, Tom Henderson 

Center for Teaching, Learning, and Technology, Washington State University 

Box 644550, Pullman, Washington 99164-4550 USA 

dbennet@mail.wsu.edu 

browng@wsu.edu 

tom@wsu.edu 

Abstract 

New educational technologies and online learning environments (OLEs) are infiltrating today’s college 

classes and campuses. While research has examined many aspects of this permeation, one research gap 

exists. How do faculty perceive the learning experience in courses that use OLEs compared to courses that 

do not? One important factor that may influence faculty perceptions are their reasons for teaching with 

OLEs. This paper seeks to understand how faculty perceive OLEs as a function of their reasons for teaching 

with this educational technology. This paper also investigates whether faculty evaluations of OLEs differ 

based on gender and by years teaching. The results of the analysis reveal several noteworthy patterns. First, 

it appears that favorable opinions about the learning experiences in online learning environments are not 

because faculty are motivated to learn about new technologies per se, but because they want to update their 

vitas and teaching skills. Second, the results suggest that it may be harder to convince older and more 

experienced faculty to use new technologies compared to younger and less experienced faculty. These 

results apply to both male and female faculty and provide practical implications for universities and support 

services on how to recruit and then support faculty who implement educational technologies. 

Keywords 

Higher education, Learning technologies, Online education 

Background 

Debates arise at our higher education institutions about the value of educational technologies and online learning 

environments (OLEs) as this technology infiltrates our classrooms and demands persist that our students become 

technologically literate. The push for a “wired” campus has become the norm. At the same time, concerns about 

teaching with technology in online spaces continue apace as the production of untested educational technologies 

escalates. Over the past 11 years, the Campus Computing Project has administered an annual survey to measure 

the relative importance of technology use on college campuses (Carlson, 2000). Over 500 private and public 

institutions have participated in the survey, including both two- and four-year colleges. Data from the surveys 

indicate that the use of technology in instruction has risen sharply in college courses since 1994. Further, 40 

percent of the senior information-technology administrators who responded to this year’s survey reported that 

incorporation of technology into the classroom was “the single most important issue” over the next two to three 

years. Incorporating technology into the classroom outranked other challenges, such as user support, replacing 

outdated hardware or software, providing distance education online, and integrating e-commerce on campus web 

sites (Carlson, 2000). 

The rush to incorporate technology into college courses and the increasing use of OLEs present many crucial and 

pressing issues that research must address. The foremost of these issues is to understand the educational 

advantages and disadvantages of OLEs. For student outcomes, current research is ambiguous and finds mixed 

results. In a review of extant research, Ehrmann (1995) finds both positive and negative impacts of educational 

technologies on student learning. For example, computer-based tutorials improved learning outcomes by 20%, 

but the lag in time between the development and implementation of the software often means that the software is 

dated before it can be widely used. In a comparative study of instruction between online and face-to-face 

learning environments, Johnson et al. (2000) find no significant differences in learning outcomes or student 

satisfaction. A review by Moore & Kearsky (1996) also concludes that there is no real evidence that technology 

enhances student learning. Overall, Clark (1994) insists that technology and OLEs have no influence on student 

learning. He maintains that any increases in student learning are not due to technology per se, but to the teaching 






78

method built into the use of the technology. Perhaps the lack of significant findings is because current research 

on technology and student learning is hampered in several ways. For example, the research base to inform 

practice is small and few theoretical frameworks guide current research (Menges & Austin, 2001). 

Regardless, Chickering & Ehrmann (1996) argue that technology and OLEs may assist in the implementation of 

the seven principles for good practice in undergraduate education. In this sense, OLEs benefit both faculty and 

student development by advancing specific acts of teaching and learning such as (a) contact between students 

and faculty, (b) reciprocity and cooperation among students, (c) active learning techniques, (d) prompt feedback, 

(e) time on task, (f) communication of high expectations, and (g) respect of diverse talents and ways of learning. 

For example, the use of online instruction, email, and bulletin boards encourage student-faculty contact by 

increasing the accessibility of faculty, sharing of resources, and ease in which faculty and students can “safely” 

discuss controversial issues (Chickering & Ehrmann, 1996; Graham et al., 2001). 

In a series of review articles, Emerson & Mosteller (1998a, 1998b) find that computer-assisted instruction is 

associated with positive student attitudes and shortened instructional time. They also find that the use of 

computer software is linked to learning gains and increased learning efficiency, but are still not replacements for 

creative and dedicated teachers. 

For faculty, OLEs also present advantages and disadvantages. McInnis (2002) argues that OLEs increase the 

opportunities for faculty to organize student groups, instruct students and support student learning, and evaluate 

student performance. Yet, the innovations in technology that promote these opportunities also increase 

substantially the time and energy required of faculty to achieve the potential of OLEs. Further, the rapid 

proliferation of educational technologies and OLEs change the criteria by which faculty performance is judged. 

Faculty are now evaluated on the extent to which they use technologies and whether this usage is effective in 

improving student learning outcomes (McInnis, 2002). Nonetheless, Gilbert (1996) contends that faculty who 

experiment with OLEs undergo a conversion experience that makes them better teachers by encouraging 

reflection on teaching approaches and by initiating dialogue with colleagues on the merits of different teaching 

approaches. 

To be sure, then, extant research on the effects of educational technology and OLEs is mixed, at best. An underresearched 

topic that can contribute to the debate about the advantages and disadvantages of OLEs is the 

perceptions of faculty who use educational technologies and teach in OLEs and the reasons why they use these 

tools. Understanding why faculty use OLEs and whether they perceive any educational benefits associated with 

OLEs will help frame other research that examines student learning vis-à-vis educational technologies. 

Overall, Nantz & Lundgren (1998) find that faculty are limited in exploiting the potential of new technologies. 

Olcott & Wright (1995) note that faculty are reluctant to use new technologies despite the growing trend of more 

courses offered online and the use of online technologies, but the reasons for this are somewhat elusive. Several 

possible reasons that may explain a faculty member’s reluctance to teach with online technologies include lack 

of instructional support, increased workloads, and lack of monetary compensation (Carr, 2000; Northrup, 1997; 

Thompson, 2002). Other reasons may include the relative novelty and ambiguity about the benefits of 

educational technologies. Indeed, the current teaching practices of most college faculty represent a more 

traditional approach to teaching (e.g., lectures, class discussions), which generally occur inside the classroom 

and are monitored directly by the faculty (Willcoxson, 1998). These practices differ from those involved with 

using technology in OLEs, which occur outside the classroom and the direct monitoring by the faculty. 

However, Thompson (2002) reports that faculty generally appear to be enthusiastic and satisfied about the 

benefits of and experiences with online educational technologies even though they identify several possible 

drawbacks. The largest teachers’ union, the National Education Association, released the findings from their 

survey on higher education members’ attitudes about distance learning. Data from 532 college faculty reveal that 

three-quarters of the instructors report a positive outlook about online learning (Carr, 2000). 

Olcott & Wright (1995) suggest that one way to support faculty who use educational technologies in OLEs is 

through instructional support. Instructional support includes instructional design, course developers, technical 

support, and teaching and learning specialists (Lee, 2001). Indeed, faculty who change their professional 

behaviors must feel a sense of ownership of their new practices (Wood & Thompson, 1993). This suggests that 

the usefulness of change and using educational technologies may depend on an individual faculty member’s 

professional control in the way this technology is incorporated into their existing teaching repertoire. It is 

important that faculty view these technologies as a legitimate and effective component of teaching and learning. 

Indeed, others suggest that faculty may be more motivated to teach with educational technologies in online 

spaces if the activities are associated with teaching rather than with extrinsic or monetary rewards (Peirpoint & 

Hartnett, 1988; Taylor & White, 1991; Wolcott & Betts, 1999). 

79

Another mechanism to support faculty who use educational technologies is to demonstrate the benefits of such 

technologies. Research shows that teachers’ perceptions about new instructional practices influence their 

teaching decisions regarding instruction and curriculum and these beliefs affect subsequent student performance 

(Ennis, 1998; Ross, 1998). That is, for effective teaching and learning, it is not enough that faculty have 

knowledge about educational technologies, but that they believe in the effectiveness of that technology and, in 

turn, use the technology resourcefully (Creed, 1997; O’Donnell, 1994). 

Taken together, this research suggests the following proposition: The reasons why faculty use educational 

technologies and OLEs may influence a faculty’s perception of the learning experiences associated with such 

instructional practices. For example, faculty who embrace the opportunity to use educational technologies may 

regard the new technologies more positively than faculty who were simply asked to teach an existing course that 

used educational technologies. Likewise, faculty with a genuine interest in learning about new technologies may 

perceive the learning opportunities in OLEs more positively than faculty who use technologies for lesser 

pragmatic reasons (e.g., to make additional money). In addition, younger faculty who may have more experience 

with educational technology may hold more favorable views than older faculty who have less exposure to and 

training with educational technologies. 

Overall, understanding the relationship between faculty reasons for using educational technologies and their 

perceptions of the learning experiences of educational technologies may shed light on how best to assist faculty 

in developing successful teaching strategies that incorporate educational technologies. Further, understanding 

how the various factors that influence faculty’s decisions to teach using new technologies and how these factors 

shape their perceptions of OLEs can help universities with a smoother transition to the wired campus and result 

in a positive influence on student learning. 

Present Study 

This study seeks to understand the relationship between (a) faculty perceptions of the learning experience in 

courses that use OLEs and (b) their reasons for teaching and using OLEs. This paper also investigates whether 

faculty perceptions differ based on gender and years teaching. This study may inform faculty-mentoring and 

development programs on how to support faculty who use or plan to use educational technologies by aligning 

the incorporation of technology into their classrooms with their reasons for teaching with educational 

technologies. Technologies that assist online learning are relatively new, even to young faculty who recently 

finished graduate school. Therefore, it is important to examine how traditional and nontraditional reasons for 

teaching are associated with faculty’s perceptions. For example, if traditional reasons for teaching are associated 

negatively with perceptions of the learning experiences in courses that use OLEs, then it is imperative to identify 

ways this new technology can fit well within a traditional educational setting that employs traditionally-trained 

faculty. 

Sample 

Data for this analysis come from The Center for Teaching, Learning, and Technology (CTLT) at Washington 

State University (WSU) as part of an ongoing assessment process developed to systematically evaluate the use 

and impact of innovative teaching practices. The CTLT, WSU faculty, and other educational professionals have 

developed a series of surveys (available on request) that focus on faculty and student learning goals, activities, 

and practices (GAPs). The GAPs survey process involves three surveys—one for faculty and two for students. 

The surveys were distributed online via a survey generator (CTLSilhouette) developed by the CTLT. All faculty 

using the centrally supported online learning technologies at WSU were invited to participate in the GAPs 

survey process. This study uses data from the Fall 2000 and Spring 2001 faculty surveys. The final sample 

represents a cross-section of 85 faculty, including 30 faculty who augmented their campus-based course with 

OLEs, 25 faculty who teach students at a distance with OLEs, and 25 faculty from other higher education 

institutions who use WSU-supported OLEs. 

Dependent Variable: Usefulness of Online Learning Technologies 

One question measured a faculty’s perception of students’ learning experiences in OLEs. The question was 

worded: “In your opinion, a student’s learning experience in a course that uses online learning environments 

compared to a course that does not is, overall…” The possible response categories were coded so higher scores 

80

epresent evaluations that are more positive: 1 = can never be as good, 2 = usually worse, 3 = slightly worse, 4 = 

about the same, 5 = slightly better, 6 = usually better, 7 = always better. 

Independent Variables: Reasons for Teaching with Educational Technologies 

The surveys asked faculty members why they opted to teach courses that use OLEs. The final set of reasons that 

appeared on the surveys was generated from numerous faculty focus groups and advice from educational 

specialists. Specific wording of the question addressing their reasons for teaching with OLEs was: “How 

important to you are the following reasons for teaching this course using online learning environments?” The 

reasons addressed were: (1) To teach nontraditional students with work experience, (2) I was asked to teach this 

course, (3) To learn or stay abreast of new educational technologies in the classroom, (4) To support program 

goals of my department, college, or school, (5) To make additional money, (6) It makes sense to use an online 

learning environment in my course (e.g., the course may be about new technologies), (7) To develop new 

teaching skills, and (8) To improve my vita or resume. The possible response categories were coded so higher 

scores represent greater importance: 1 = not important at all, 2 = not very important, 3 = somewhat important 

and 4 = very important. 

Two demographic variables were used in the analyses: sex of the faculty (1 = female; 0 = male) and years of 

teaching experience (1 = first class, 2 = 0 to 2 years, 3 = 2 to 5 years, 4 = 5 to 10, and 5 = 10 or more years). 

Data Issues 

The data came from questions that were part of the GAPs survey emailed to the instructors. A random sample 

was not feasible because of the nature of the investigation. The data, thus, were generated from a convenience 

sample. However, statistical analyses on the distribution of the independent and dependent variables revealed no 

deviations from normality or clustering of responses. Further analyses indicated the faculty who responded were 

not from selective colleges, disciplines, or departments (results available on request). Admittedly, the nonrandom 

sample limits the ability to generalize to the larger population of college faculty. The results presented here, 

therefore, will be most useful when compared in context to other existing studies on the perceptions of faculty. 

Analytic Strategy 

Ordinary least squares regression was used to estimate the effects of the reason-for-teaching variables on faculty 

reports of students’ learning experience in courses that use OLEs. The first set of equations estimated the 

bivariate associations by regressing separately faculty evaluations on each predictor variable. The second 

equation regressed faculty perceptions on all of the predictor variables in one full model. The full model 

estimates the net effect of each reason-for-teaching variable while controlling for the other reasons for teaching. 

The two demographic variables are included in both analyses. 

Scores on the dependent variable range from 1 – 7. Because of the debate whether these Likert-type scales 

represent ordinal or interval data, the equations were reestimated using both ordered and multinomial logistic 

regression methods (Long, 1997). The results were identical to those for OLS regression. The OLS regression 

results are presented because the substantive interpretation of the parameter estimates, including R-square, is 

better known. 

Results 

Descriptive data from Table 1 indicate that women comprise 65% of the sample and that the average faculty 

member has taught between 5-10 years. On average, faculty report a moderately positive evaluation of the 

learning experience in courses that use OLEs compared to courses that do not use OLEs—5.32 on a 7-point 

scale. Looking at the reasons for teaching courses that use OLEs, responses from the faculty show that three 

reasons were ranked the highest: to develop new teaching skills, to learn or stay abreast of new technologies and 

try them in the classroom, and to support program goals of my department or college. Two reasons for teaching 

were ranked consistently low by faculty: to make additional money, and because they were asked to teach the 

course. 

81

Table 1: Summary of Descriptive Statistics of Study Variables 

Variables and Coding Mean 

Dependent 

Perception of learning experience in courses that use OLEs versus courses that do 

not (1 = Can Never be as Good to 7 = Always Better) 

5.32 1.16 

Predictors (1 = Not Important at All to 4 = Very Important) 

To teach nontraditional students or students with work experience 2.53 1.19 

I was asked to teach this course 2.15 1.02 

To learn or stay abreast of new technologies and try them in the classroom 3.34 .82 

To support program goals of my department or college 3.29 .81 

To make additional money 1.84 1.10 

It makes sense to use an online learning environment in my course 2.30 1.03 

To develop new teaching skills 3.46 .71 

To improve my vita 2.27 .92 

Demographic 

Years teaching (1 = first class to 5 = over 10 years) 4.16 1.15 

Sex (1=female, 0=male) .65 .47 

Standard 

Deviation 

How do faculty’s reasons for teaching affect their evaluation of the relative learning experience in courses that 

use OLEs? The bivariate regression coefficients in the first column of Table 2 show that faculty’s perceptions of 

OLEs is significantly greater when they were more likely to teach a course (1) To learn or stay abreast of new 

technologies and try them in the classroom, (2) It makes sense to use an online learning environment in the 

course, (3) To develop new teaching skills, and (4) To improve my vita. 

Table 2: Results of Regression Analyses 

Predictor Variables Bivariate Multivariate 

To teach nontraditional students or students with work experience .007 

(.107) 

R 2 .032 

= .000 

(.108) 

I was asked to teach this course -.022 

(.124) 

R 2 -.074 

= .000 

(.124) 

To learn or stay abreast of new technologies and try them in the classroom .196** 

(.074) 

R 2 -.127 

= .077 

(.090) 

To support program goals of my department or college .056 

(.157) 

R 2 -.010 

= .002 

(.151) 

To make additional money -.191 

(.114) 

R 2 -.410*** 

= .033 

(.125) 

It makes sense to use an online learning environment in my course .313** 

(.119) 

R 2 .337** 

= .078 

(.107) 

To develop new teaching skills .749*** 

(.159) 

R 2 .595** 

= .212 

(.200) 

To improve my vita .345** 

(.134) 

R 2 .342* 

= .074 

(.151) 

Years teaching -.275** 

(.107) 

R 2 -.314** 

= .074 

(.098) 

Sex .182 

(.269) 

R 2 -.175 

= .006 

(.240) 

82

Intercept R 2 

Note: Standard errors are in the parentheses. *p < .05. ** p < .01. ***p < .001 (two-tailed) 

4.845 

.426 

The largest effect on a faculty’s perception of OLEs occurs when faculty’s reason for teaching the course is “to 

develop new teaching skills.” This variable alone explains 21% of the variance. A coefficient of .749 suggests 

that a faculty’s evaluation of the relative learning experience in courses that use OLEs increases nearly threefourths 

of a unit for each unit increase in the importance of the reason for teaching the course. For example, the 

difference in evaluations would be 1.50 between faculty who feel it is very important to develop new teaching 

skills (coded 4) and faculty who feel it is somewhat important to develop new teaching skills (coded 2). The 

effects of the other three significant variables are smaller but all explain at least 7% of the variance in faculty’s 

evaluations of OLEs. 

On the other hand, faculty’s perceptions of the learning experiences in OLEs are significantly lower the more 

years they have been teaching. Each unit increase in years of teaching experience is associated with a decrease of 

.275 in a faculty’s evaluation of OLEs. For example, the difference between faculty with over 10 years of 

experience (coded 5) and faculty with 0 to 2 years (coded 1) of experience is -.875, or about a one-unit decrease. 

Differences in teaching experience explain 7.4% of the variance in perceptions. 

The multivariate equation in Table 2 estimates whether any single reason for teaching has an effect on 

perceptions of the learning experience in courses that use OLEs compared to courses that do not, after 

controlling for the other possible reasons for teaching. In a sense, the multivariate model is a more accurate 

estimation of the relationships between reasons for teaching and perceptions of OLEs because faculty generally 

reported several important reasons for teaching. Thus, results from the multivariate regression equation will 

better simulate this reality by including simultaneously all of the important reasons for teaching. Overall, the 

multivariate results shed new light on the bivariate relationships and shows that reasons for teaching are a 

significant component of a faculty’s perception of the learning experience associated with OLEs. Indeed, an Rsquare 

of .426 suggests that nearly 43% of the variance in faculty’s perceptions of online learning environments 

is explained by their reasons for teaching and selected demographic characteristics. 

Four of the statistically significant bivariate effects remain so in the multivariate model. Faculty’s perceptions on 

the learning experience of OLEs is significantly more positive when they were more likely to teach the course 

because (1) It makes sense to use an online learning environment in the course, (2) To develop new teaching 

skills, and (3) To improve my vita. Faculty evaluations of OLEs are significantly lower the more years they have 

been teaching. However, two new patterns emerge in the multivariate results. First, faculty are more likely to 

report more positive perceptions of OLEs if they place little value on making additional money as a reason for 

teaching courses that use OLEs. This suggests that when all the other reasons for teaching are controlled, there 

exists a statistically significant negative relationship between money as a reason for teaching and perceptions of 

OLEs. 

Second, the net effect of staying abreast of new technologies as a reason for teaching becomes statistically 

nonsignificant between the bivariate and multivariate regression models. This indicates that one or more of the 

other reasons for teaching explain the prior significant bivariate effect of technology. In ancillary analyses 

(results available on request), we estimated a series of regression models to determine which other variable(s) 

account for the bivariate effect of technology. Results show that the effect of technology remains significant until 

both “To develop new teaching skills” and “To improve my vita” are in the regression equation. What does this 

mean? While not definitive, it suggests that the effect of staying abreast of new technology on a faculty’s 

perception of OLEs is indirect and may operate through the other two variables. That is, part of the reason why 

faculty learn and implement new technologies is so they can improve both their teaching skills and vita. 

Discussion 

Educational technologies are playing an increasingly larger role in college courses, and faculty members are 

beginning to implement these technologies into their classrooms and into online learning environments. The 

incorporation and effective implementation of online learning technologies may depend on how individual 

faculty perceive the merit of such technologies and how the use of these technologies fit with their current 

teaching practices and beliefs. The results of the present analysis reveal several noteworthy patterns. First, it 

appears that favorable evaluations about the learning experiences in courses that use OLEs are not because 

faculty necessarily want to learn about these new technologies per se, but because faculty wish to supplement 

83

and update their vitas and improve their teaching skills. This has practical implications for how universities 

might recruit and then support faculty who teach with OLEs. Instruction on teaching with new technologies may 

be imbedded in workshops, seminars and programs that target career and professional development and that 

focus on improving student learning. Conversely, workshops that target technology in isolation may be less 

effective in encouraging faculty to use OLEs. 

Why would universities want to recruit and train faculty to use educational technologies and teach courses that 

use OLEs? Several reasons emerge, including the absolute acceleration of such technologies, but also because 

educational technologies may open new avenues for more students to access opportunities and information, 

increase forms of interactions among teachers and students, and encourage collaboration across institutions 

(Menges & Austin, 2001). As the demographics of college students continue to diversify, universities and 

colleges have also realized that OLEs such as distance education and online instruction can reap financial 

benefits, and the recruitment of quality faculty to staff these types of courses can improve an institution’s bottom 

line. 

Second, the results suggest that older and more experienced faculty hold less favorable opinions on the learning 

experiences associated with courses that use OLEs compared to courses that use more traditional instructional 

practices. These results apply to both male and female faculty. This finding has relevant implications for policy 

and practice, suggesting that the recruitment and development process requires a considered and careful 

coordinated response to faculty readiness and unique purposes for teaching and learning. If more experienced 

faculty are to use educational technologies, they may need solid evidence about the effectiveness of such 

technologies. For these faculty, additional feedback, support, and training may increase their likelihood of using 

educational technologies. 

This study confirms previous findings that faculty will employ new instructional practices if they are based on or 

enhance general acts of teaching. This study provides new evidence that these reasons are associated with their 

evaluations about the learning experience in courses that use OLEs. Ehrman (1995), Guskin (1994), and McInnis 

(2002) all argue that the role of technology in higher education should be to enhance acts of teaching—e.g., 

prompt feedback, faculty-student interaction, transmission of information, and development of active learning 

opportunities—that, in turn, enhance student learning. Indeed, Wolcott & Betts (1999) report the results of 

survey and interview data from about 600 faculty and find three interesting trends that support this interpretation. 

First, faculty reported that using educational technologies and OLEs required significant amounts of extra time 

and preparation—so-called “hidden work.” Second, to take on this extra work, faculty were not driven by 

external factors such as merit pay or promotions. Instead, and third, faculty choose to spend extra time and use 

OLEs to increase the opportunities to grow professionally. Even though many faculty were intrigued by 

technology, they viewed it more as a means to improve their teaching practices and overall student learning. 

Future research on education and technology could extend this study in one important way. Specifically, do the 

reasons why faculty teach courses that use OLEs predict a quality learning environment? This is a different 

question than that of perceptions of the learning experience. More generally, all research on education and 

technology must become more comprehensive by including both faculty and students (Menges & Austin, 2001). 

Indeed, with an eye toward faculty and students, MacFarlane (1995) contends that the primary goals for faculty 

who use technology are to (a) structure knowledge to make it interactively accessible, (b) facilitate the process of 

learning, and (c) manage interaction among learners and between learners and knowledge. Finally, Saba (2000) 

argues that comparative studies that examine the effects of educational technology on student learning must be 

grounded in theories of learning and interaction. 

References 

Carlson, S. (2000). Campus survey finds that adding technology to teaching is a top issue. The Chronicle of 

Higher Education, 47, A46. 

Carr, S. (2000). Many professors are optimistic on distance learning, survey finds. The Chronicle of Higher 

Education, 46, A35. 

Chickering, A. & Ehrmann, S. C. (1996). Implementing the seven principles: Technology as a lever. AAHE 

Bulletin, 48, 3-6. 

84

Clark, R. E. (1994). Media will never influence learning. Educational Technology Research & Development, 42, 

21-29. 

Creed, T. (1997). Extending the classroom walls electronically. In Campbell, W. E & Smith, K. A. (Eds.), New 

paradigms for college teaching, Edina, MN: Interaction Book Company, 149-184. 

Johnson, S. D., Aragon, S. R., Shaik, N. & Palma-Rivas, N. (2000). Compararive analysis of learning 

satisfaction and learning outcomes in online and face-to-face learning environments. Journal of Interactive 

Learning Research, 11, 29-49. 

Ehrmann, S. C. (1995). Asking the right question: What does research tell us about technology and higher 

education? Change, 27, 20-27. 

Emerson, J. D. & Mosteller, F. (1998a). Interactive multimedia in college teaching. Part I: A ten-year review of 

reviews. Educational Media and Technology Yearbook, 23, 43-58. 

Emerson, J. D. & Mosteller, F. (1998b). Interactive multimedia in college teaching. Part II: Lessons from 

research in the sciences. Educational Media and Technology Yearbook, 23, 59-75. 

Ennis, C. D. (1998). Shared expectations: Creating a joint vision for urban schools. In Brophy, J.(Ed.), Advances 

in research on teaching: Expectations in the classroom, Vol. 7, Greenwich, CT: JAI Press, 151-182. 

Gilbert, S. (1996). Making the most of a slow revolution. Change, 28, 10-23. 

Graham, C., Cagiltay, K., Craner, J., Lim B. & Duffy, T. M. (2000). Teaching in a web-based distance learning 

environment: An evaluation summary based on four courses. Center for Research on Learning and Technology 

Technical Report No. 13-00. Bloomington, IN: Indiana University. 

Guskin, A. E. (1994). Restructuring the role of faculty. Change, 26, 16-25. 

Lee, J. (2001). Instructional support for distance education and faculty motivation, commitment, satisfaction. 

British Journal of Educational Technology, 32, 153-160. 

Long, J. S. (1997). Regression models for categorical and limited dependent variables. Thousand Oaks, CA: 

Sage. 

MacFarlane, A. G. J. (1995). Future patterns of teaching and learning. In Schuller, T. (Ed.), The changing 

university, Bristol, PA: Open University Press, 12-38. 

McInnis, C. (2002). The impact of technology on faculty performance and its evaluation. New Directions for 

Institutional Research, 114, 53-61. 

Menges, R. J. & Austin, A. E. (2001). Teaching in higher education. In Richardson, V. (Ed.), Handbook of 

research on teaching, 4 th Edition, Washington, DC: American Educational Research Association, 1122-1156. 

Moore, M. & Kearsky, G. (1996). Distance education: A systems view. Belmont, CA: Wadsworth. 

Nantz, K. S. & Lundgren, T. D. (1998). Lecturing with technology. College Teaching, 46, 53-56. 

Nothrup, P. T. (1997). Faculty perceptions of distance education: Factors influencing utilization. International 

Journal of Educational Telecommunications, 3, 343-358. 

O’Donnell, J. J. (1994, May 18-20). Where are we going and when did we get there? Presentation at the 124 th 

annual meeting of the ARL: The research library the day after tomorrow. Austin, TX. 

Olcott, D. & Wright, S. J. (1995). An institutional support framework for increasing faculty participation in 

postsecondary distance education. American Journal of Distance Education, 9, 3, 5-17. 

Pierpoint, P. E. & Hartnett, R. A. (1988). Faculty attitudes towards teaching in off-campus graduate programs. 

International Journal of Innovative Higher Education, 5(1), 25-30. 

85

Ross, J. A. (1998). The antecedents and consequences of teacher efficacy. In Brophy, J. (Ed.), Advances in 

research on teaching: Expectations in the classroom, Vol. 7, Greenwich, CT: JAI Press, 49-74. 

Saba, F. (2000). Research in distance education: A status report. International Review of Research in Open and 

Distance Learning, 1, 1-9. 

Taylor, J. C. & White, V. J. (1991). Faculty attitudes towards teaching in the distance education mode: An 

exploratory investigation. Research in Distance Education, 7-11. 

Thompson, M. (2002). Faculty satisfaction. Sloan-C View, 1, 6-7. 

Willcoxson, L. (1998). The impact of academics' learning and teaching preferences on their teaching practices: A 

pilot study. Studies in Higher Education, 23, 59-70. 

Wolcott, L. L. & Betts, K. S. (1999). What's in it for me? Incentives for faculty participation in distance 

education. Journal of Distance Education, 14, 34-49. 

Wood, F. H., & Thompson, S. R. (1993). Assumptions about staff development based on research and best 

practice. Journal of Staff Development, 14, 52-57. 

86

Bozarth, J., Chapman, D. D., & LaMonica, L. (2004). Preparing for Distance Learning: Designing An Online Student Orientation 

Course. Educational Technology & Society, 7 (1), 87-106. 

Preparing for Distance Learning: Designing An Online Student Orientation 

Course 

Jane Bozarth 

NCOffice of State Personnel 

E-Learning Coordinator 

101 W. Peace Street Raleigh, NC 27613 USA 

919-733-8339 

jbozarth@ncosp.net 

Diane D. Chapman 

Department of Adult and Community College Education, North Carolina State University 

Visiting Assistant Professor 

310 Poe Hall, Campus Box 7801 Raleigh, NC 27695-7801 USA 

919-513-4872 – phone 

919-515-6305 - fax 

diane_chapman@ncsu.edu 

Laura LaMonica 

D. P. Associates, Inc. 

Instructional Systems Designer 

686 W. Corbett Ave, Suites 3&4 Swansboro, NC 28584 USA 

llamonica@hotmail.com 

Abstract 

This paper describes the analysis undertaken to design a 1-credit-hour online orientation course for students new 

to online learning. An instructional design team, as a part of an advanced instructional design course, worked 

with a university-based client. The client identified specific problem areas encountered by novice students of 

online courses and the team designed a comprehensive program to meet those needs. Analysis of the data 

revealed surprising differences in expectations between instructors of online courses and their students of what 

an orientation to online learning should include. The team also conducted a task analysis to aid in further 

identifying the skills, knowledge and attitudes required by students for success in online courses. Findings 

indicated that there is a need for online learners to understand the time commitment required of an online course 

and possess or develop strong time management skills. Because of small sample size, results cannot be 

generalized beyond the respondents. The authors found a mismatch in the perception of instructor technical 

skills versus student technical skill. Based on their findings, the paper provides recommendations on the 

appropriate design, development and implementation of an orientation to online learning. 

Keywords 

Online learning, Distance learning, Skills assessment, Task analysis, Instructional design 

Training Situation 

Introduction 

As a requirement of an advanced instructional design course at North Carolina State University (NCSU), a group of 

students embarked on a project to design an online orientation course for new online learners. The project team 

consisted of four students, all employed as training professionals by commercial or government organizations. The 

client was a school within a local university that had been in the distance-learning arena for some time. The client 

had specific requirements for the course and delineated needs that the course was to address. The outcome of the 

project was to be a pre-requisite course to prepare new students for the online educational environment. 

Emphasizing a need to focus on issues besides just the technical, the client identified four areas as potential topics to 

include in this online course: 





specific permission and/or a fee. Request permissions from the editors at kinshuk@massey.ac.nz. 

kinshuk@ieee.org. 

87

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

Setting appropriate expectations 

Guidance in online etiquette 

Information on available support resources 

An assessment of the readiness of the student for online learning. 

Reason for Study 

Typically, the client university has a "hit or miss" approach to preparing students for online learning. Often these 

orientation meetings consist of short face-to-face sessions or sessions in which students must share computers. In 

their request, the client identified several key problems resulting from their current orientation methods supporting 

their conclusion that the school needs a course that focuses on more than the technical training. These problems 

include: 

Students are not trained properly because the training is more an afterthought. 

Students cannot apply the lessons directly with their home or office computers. 

Training approaches lack consistency and completeness. 

Students cannot request assistance with configuration issues with their computers. 

Students cannot test the technology in a realistic setting. 

The approach does not reach beyond campus and limits the number and type of students served. 

Instead of technical training, the client was looking for a course that orients students to the online learning 

environment, including managing the unfamiliarity of an online course management interface. 

Additionally, a review of the literature revealed support for development of an online orientation course. Online 

learning requires different skills and talents than the classroom setting, from new communication patterns to more 

finely honed time management skills. Berge (2001) and Willis (1992) discuss the different skills demanded of online 

learning versus the more familiar classroom, from new communication patterns to more finely-honed time 

management skills, and stress the need for students to be adequately oriented to the new online environment in order 

to facilitate their success. 

Main Assumptions 

Upon reviewing the client’s request and discussing the needs with the client, the project team concluded that a 

training course was indeed needed. The team also recognized that such a course would be beneficial to other 

University of North Carolina (UNC) system schools as well. In particular, the course could be of significant interest 

to the Department of Adult and Community College Education (ACCE) at NCSU, as that department is about to 

begin a degree program completely online beginning in the fall of 2002. 

Study Objectives 

A study was proposed to determine the content and structure of the orientation course. The project team's goal was to 

utilize survey instruments to: 

Identify the expectations instructors had of students entering an online course. 

Identify the expectations students had of what an online course would entail and how the instructor would 

manage the course. 

Determine whether gaps exist between instructor and student expectations of online learning. 

Identify the specific topics that the instructors and students deemed as important to include in an online prerequisite 

course. 

Identify the key problem areas for students in online courses. 

Identify frequently asked questions that online instructors receive from students. 

Identify the average student dropout rate from online courses and the reasons for attrition. 

Identify how many hours per week instructors typically expect students to devote to an online class. 

88

Scope of Study 

Based on client input, the team concluded at the outset of the analysis that this course is not intended to sell online 

learning to wavering students or to instill in students, basic computer literacy. Given that parameter, the project team 

designed survey questions to uncover key topic areas that the students and instructors might identify to include in the 

training program design. The questions were designed to also elicit information about students' assumptions about 

online learning. Understanding these assumptions will help the team to prioritize topics and add additional 

information during the design phase to dispel any incorrect assumptions that potential students may have. 

Additionally, the team felt that designing a questionnaire specifically geared toward instructors would uncover 

students' limitations with online learning and offer suggestions to aid in the course design. 

The project team identified several target audiences to survey: 

The client’s instructors (with and without experience in online teaching) 

NCSU instructors (with and without experience in online teaching) 

The client’s students (with and without experience in online learning) 

NCSU students (with and without experience in online learning) 

Students currently participating in online courses at University of Phoenix 

NCSU and client students and instructors were key groups from whom the project team wanted to receive feedback. 

These groups would provide firsthand accounts of their experiences, assumptions and beliefs associated with online 

learning. The team chose another e group from which to receive feedback: University of Phoenix online students. 

The team hoped these respondents would "round out" the survey responses and allow for triangulation of the data. 

The University of Phoenix group was comprised of individuals from a local employer that were currently enrolled in 

online courses via the University of Phoenix. This provided the team with a benchmark to which to compare the 

client/NCSU online learning system. This information strengthened the team's perspective and position and guided 

us in the design of some of our core introductory topics. 

Upon reviewing the survey data, the project team agreed that a need had been identified and began designing the 

course to orient students to the online learning environment. 

Literature Review 

In approaching course design the project team also looked at current literature on distance learning. Berge (2001) 

supports the notion of some sort of “orientation” for new distance learners: “Instructors have a right to expect that 

participants will come to distance learning experiences prepared to study effectively at a distance…(such materials 

as)…a student handbook, a preliminary screening survey, or even a mini-course that would help ensure that learners 

acquire appropriate study and learning skills and understand their rights and responsibilities in a distance learning 

course” (p. 20-21). Willis (1992) echoes this: “Make students aware of and comfortable with new patterns of 

communication to be used in the course…Assist students in becoming familiar and comfortable with the delivery 

technology and prepare them to resolve the technical problems that will arise” (p. 3). 

The research on learner characteristics supports what was found in the survey of NCSU and UNC students. Hardy 

and Boaz (1997) surveyed 200 academic distance learners and found that the students, in identifying factors for 

success, felt that they needed to be more independent, assertive, self-disciplined and motivated than the average 

college student. Additionally Sherry (1996), citing Charp, writes that crucial characteristics include active listening 

and the ability to work independently in the absence of a live instructor. Citing Brent and Bugbee’s (1994) study, 

Sherry (1996) reports that students who passed their courses “differed significantly in primary strategies from those 

who failed: in testwiseness, concentration, and time management skills” (p.10). On the issue of time management, 

Mason (2001) describes time as “the new distance” and argues that lack of time, rather than problems arising from 

distance, has become one of the primary reasons for student dropout. 

The assessment team felt that the review of literature supported the need for an orientation to online learning course, 

as well as content geared toward enhancing technological and self-management skills, while providing a realistic 

image of the online learning experience. 

89

Situation Analysis 

Methodology 

Students and instructors were interviewed via questionnaire. Respondents may or may not have had experience in 

online courses. Questionnaires were developed to include both closed- and open-ended questions to gather a variety 

of feedback. The team worked jointly to formulate questions based on their own experiences as students in online 

environments. Referring to the issues originally identified by the customer and issues team members themselves 

experienced and encountered as online students, the team was able to settle on key areas about which to construct 

questions, including technical skills, assumptions about online learning, and challenges of online learning. 

Questionnaires were distributed to one group of students at a time, usually the members of an online course and, as 

results were returned, the team was able to refine the questionnaire to elicit more useful and complete data. Student 

questionnaires are found in Appendix A. 

Questionnaires for all audiences were distributed in a variety of ways. Student questionnaires were posted directly to 

the discussion forum of online classes or were distributed by an online instructor to other online classes. Instructor 

questionnaires were e-mailed to selected instructors willing to participate. Questionnaires targeted to University of 

Phoenix students were distributed by e-mail to respondents. 

A focus group was conducted with online instructors to gather feedback regarding experiences in teaching online. 

The purpose of the group was to elicit more discussion and insight around key issues identified by instructors in the 

survey data: an emphasis on needed technical skills of students, and instructor expectations. The focus group was 

informal with team members asking questions as they arose from the conversation. Minutes of the focus group were 

recorded and discussed by the team members for data extraction. 

Limitations 

A convenience sample was used in this study. Therefore the results may not be widely representative of all distance 

learners, only for the people who responded to the instrument. Also, response rates from populations to the surveys 

were lower than expected and desired. The small sample size makes this study unable to be generalized to the wider 

population. 

Results 

Instructor Survey Results 

The team assumed, based on the client's request that new online learner performance was not in keeping with the 

expectations of their instructors. An assessment needed to be developed to determine instructor expectations of new 

online learners, instructors’ reasons for feeling those expectations were not met, and instructor expectations of 

students who had completed an introductory/orientation course such as the one proposed. Additionally, the project 

team was seeking information that would suggest the cognitive, psychomotor, and affective objectives of the course 

they were charged with developing. In creating the assessment it seemed that surveys and, time allowing, group 

interviews, would help to uncover that information. The team wrote questions targeting those specific areas, 

including: 

If a student had already completed a 1 credit hour course preparing them for online learning, what would you 

expect them to know upon entering your class? 

What are the top problem areas that students have encountered? 

What is the dropout rate for an online course? Of the students who dropped, what were their common reasons 

for dropping? 

The team also wanted to assess the particular technologies that such a course might need to address, as well as 

develop an understanding of what issues might not be within the control of the student or instructor. Thus we added 

90

the questions, "What problems are not within the students' control?" and "What frequently-asked questions do you 

receive that are not within your expertise?" Finally, all members of the assessment team are experienced online 

learners. Based on their experiences they knew that many students are surprised by the time commitment required for 

of an online course and expected this to be a major component of the final product; thus, they chose to include the 

question, "How many hours per week does a student devote to an online class?" Additionally, as 3 of the 4 team 

members felt that their experience with online learning had been adversely affected by enrolling in classes in which 

there were simply too many students, they also included the question "How many students in an online class is too 

many?" Instructor surveys are located in Appendix B. 

The survey was distributed to instructors with whom the assessment team had access: 45 instructors of online 

courses at NCSU and 8 instructors of online courses in the client institution. 17 of the 53 surveys were returned, 

resulting in a return rate of 32%. In every case the entire questionnaire was completed. No questions were left blank 

or answered "no response" or "not applicable." 

A common theme among instructor responses was the misperception among students that online courses would 

demand only that they log in once a week to get an assignment or provide a posting; instructors reported that students 

often seem surprised at the level of interaction and frequency of contact demanded by many courses. Instructors 

(Figure 1) stressed the need for students to develop a plan for completing the course, learning to be independent in 

scheduling their own time, staying on schedule, avoiding procrastination and finding ways to be efficient. 

Additionally, instructors stressed the importance of the students taking the initiative in clarifying expectations, taking 

the initiative to participate in discussions and otherwise develop relationships with other classmates, and in general 

simply learning self-management skills. 

A striking difference in the responses of students and instructors came on the issue of technology skills. 60% of 

instructors wrote, several of them strongly and at length, about the lack in many cases of what they considered to be 

basic computer skills (Figures 2 and 3). Deficits ranged from understanding how to configure browsers, using 

discussion and chat features, and using different software programs to tasks like opening PDF files or sending 

attachments. Two instructors commented that some students seem unwilling to "try out" technology before asking 

for help, or are not clear that many technical problems would be better addressed by contacting the school's help desk 

or other support resources. Three instructors commented that in some cases students, when enrolling in course, fail to 

perform the "tests" (such as sending an email to the instructor) often required in the first days of class. Finally, 5 

instructors wrote that often students simply overestimate their own technical abilities or seem to ignore the list of 

prerequisite skills required of online students. Noteworthy here is the large gap in the perceptions of the instructors 

versus those of students: based on the strength of language and length and frequency of responses, it is clear that the 

instructors perceive the technology skills deficits as a much bigger problem than do students. This will be further 

explained in the "student responses" section. 

Additionally, the team had an opportunity to interview a group of 3 instructors from NCSU's ACCE department. 

Because the team also wanted to learn more specifics about technology requirements, they invited a member of 

NCSU's computer help desk service to participate. The team based their interview format on the surveys previously 

sent out; as the instructors had already completed this written survey, the team wanted to further probe the answers 

they had given. Though on the written surveys instructors had focused primarily on technology skills, in face-to-face 

conversation they focused more on students’ need for self-discipline and time management. One instructor outlined 

detailed suggestions for success over the span of a semester: look over the course outline and identify resources 

needed—library time, reading time, trips to campus, when projects will be due, etc.—then lay out a plan for 

managing time. Another commented, "They [students] don't plan well and then, when they realize how to structure 

their time, it's too late." When asked if they felt this contributed to student dropout, all the instructors said that 

various manifestations of time management problems (students surprised 

Instructor Expectations 

Instructor Question 1: If a student had already completed a 1 credit hour course preparing them for online learning, 

what would you expect them to know upon entering your class? 

91

Specific 

Technology Skills 

14 

12 

10 

8 

6 

4 

2 

0 

13 

Instructor Expectations of New Students 

4 4 

Technology Skills Interaction Skills Understand Time 

Commitment 

How to Interact 

online 

Figure 1: Instructor Expectations 

Understanding of 

Time Commitment 

3 

Online Research 

Skills 

1 

Work 

Collaboratively 

Online Research 

Skills 

How to Work 

Collaboratively 

13* 4 4 3 1 

(*Nearly every respondent listed specific technology skills. Answers included such varied skills as using 

attachments, using spreadsheets, using graphics programs, using chat, opening PDF files, taking digital photos, 

using text-rich format, signing up for listservs, saving files, using WinZip, and converting text and graphics from 

Word and PowerPoint to HTML. Technologies specified by more than one respondent are shown in Figure 2.) 

Instructor Technology Skills 

Technology skills specified by instructors in response to Question 1 (Figure 1): 

6 

5 

4 

3 

2 

1 

0 

5 

4 

Specific Technology Skills 

Attachments Chat E-mail Download 

Documents 

3 

2 2 2 2 

Figure 2: Instructor Technology Skills 

RealPlayer Access Site Courseware 

Familiarity 

92

Attachments Chat Email Download 

Documents 

RealPlayer Access 

Course Site 

Familiarity 

with Courseware 

5 4 3 2 2 2 2 

Student Problem Areas Noted By Instructors 

Instructor Question 2: What are the top problem areas that online students have encountered? 

10 

9 

8 

7 

6 

5 

4 

3 

2 

1 

0 

9 

Student Problem Areas 

5 

Poor Technology Skills Poor Time Managemen t Poor Online Research 

Skills 

Figure 3: Student Problem Areas as Noted by Instructors 

Poor Sk ills with Particular Poor Time Management Poor Online Research Problems with 

Technology 

Skills 

Skills 

Courseware 

3 

1 

Problems with 

Courseware 

9 5 3 1 

by the amount of work required, overwhelmed by the assignments, feeling the course was just "too much 

work") 

were often at issue. They additionally said that dropout occurred because students needed credit, assumed 

an online 

course wou ld suit their needs but then found the content uninteresting and that, sometimes, students found the online 

format simply didn't suit their particular learning style. Instructors took care to say, though, that they often 

did not 

know why students dropped. The one technology skill that did generate much discussion was the subject 

of 

discussion forums, particularly in terms of the student learning appropriate "netiquette," when items would 

be more 

appropriately posted privately rather than to the group message board, and understanding that it is the 

student's 

responsibility to speak up and ask questions. One instructor said, "If you'd ask it in class, then you should 

ask it 

online." 

When asked what new online learners should expect, instructor responses included, "They should anticipate 

that their 

work will be interdependent, so they should get to know their fellow students in a variety of ways and interact 

with 

them," and "they should expect (just as with live classroom courses) that all courses will be different." When asked 

what advice they would give new online learners, one instructor said, "Define expectations with the professor 

early 

on. If 'participation' is scored, the student who is unclear about what that means should clarify it." On the 

issue of 

what an introduction to onli ne learning should cover, suggestions included providing a glossary (i.e., define the 

93

phrase "post to the discussion forum"), giving students practice time with various tools, and clarifying where—and 

when—students should turn fo r help. 

The interviewee from the NCSU help desk service 

added the input that the proposed course could not encompass 

every conceivable technological skill and, therefore, 

could not serve as "introduction to computers." He did add, 

however, 

that instructors may make too many assumptions about proficiency levels and, as he said, "They assume 

that the technology makes sense when sometimes it just doesn't." 

Student Survey Results 

The team also wanted to hear from students, particularly to learn about their expectations, versus the reality, of their 

initial experiences with online learning. Based on their own knowledge of distance learning, the team developed a 

number of questions, including: 

• If you could have learned something about online learning prior to beginning an online course, what 

would have been helpful? 

• What were your original assumptions about online learning and how have they changed? 

• What do you think is the most difficult aspect of online learning? 

• What advice would you give someone preparing to take an online course for the first time? 

The team was also interested in finding out whether students perceived the need for an introductory course and 

included the question, "If a free 1-credit hour course were offered online to help you learn more about how to be an 

online student, would you take it?" 

The resulting survey was distributed to students currently enrolled in online courses at NCSU (the 45 instructors of 


courses were asked to distribute this to their students; thus, the number of potential recipients is unknown) and 

8 at the University of Phoenix, all of whom are employees of a local pharmaceutical company. 29 surveys were 

returned. Departments represented included ACCE, Business Management, Communication; 1 respondent did not 

say with which department he was affiliated. With the exception of one respondent answering "not applicable" to one 

question, every respondent answered every question. 

Student responses dealt overwhelmingly 

with the time commitment required from an online course as shown in 

Figures 4 and 5. Seven commented 

that they had expected the online experience to be "correspondence" or self- 

study/independent 

study, where a student might perhaps read a lesson, complete an assignment, and turn it in before 

moving to the next lesson. Nine students responded that they found time management to be the most difficult aspect 

of online learning, with another 4 mentioning specifically the volume of reading required (Figure 5); when asked if 

they could have known one thing about online learning before taking their first 

Student Learning Prior To Online Course 

Student Que stion 1: If you could have learned something about online learning prior to beginning an online course, 

what would have 

been helpful? 

94

InstructorExpectations 

14 

12 

10 

8 

6 

4 

2 

0 

Time 

Commit 

ment 

Required 

13 

Instructor 

Expectations 

How to 

Post to 

Discussions 

Helpful to Learn Prior to Online Course 

5 

How to Post 

to 

Discussions 

Figure 4: Student Learning Prior to Online Course 

Knowledge 

of 

Course 

platform 

How to 

Chat 

3 

How to Chat Lack of 

Feedback 

What 

Skills I'd 

Need 

1 1 

Lack of 

Feedback 

How to 

Contact 

Classmate 

How to 

Participate 

in 

Group 

Work 

How to 

Contact 

Classmates 

13 8 5 3 3 2 1 1 1 1 

Most Difficult Aspects Of Online Learning For Students Student Question 4: What is the most difficult 

aspect of online learning? 

10 

9 

8 

7 

6 

5 

4 

3 

2 

1 

0 

9 

Ti m e 

Management 

5 

Instructor 

Inaccessibility 

Most Difficult Aspects of Online Learning 4 4 

2 2 2 

Postings Readings Timed Tests Lack 

Interaction 

Figure 5: Most Difficult Aspects of Online Learning for Students 

1 

Technology Lack Feedback 

How to 

Install 

Plug-ins 

95

Time 

Management 

/ 

Selfdiscipline 

Inaccessibility 

of 

Instructor 

Posting to 

Discussions 

Volume of 

Reading 

Timed Tests Lack of 

Interaction 

Specific 

Technologies 

9 5 4 4 2 2 2 

"Not 

knowing 

where I 

stand" 

online course, 8 said it would have been the time commitment (Figure 4); 11 wrote that the advice they would 

give a 

novice online learner would speak to the need for time management skills (Figure 6). These 11 responses 

included 

such comments as: "If you are a procrastinator, this might not be the class you want to take," and, "make 

sure you 

are person ally motivated and disciplined to do the work." Comments by students described their own failure to 

anticipate the need for near-daily check in, the need to frequently read and respond to discussion postings, 

or the 

expectation that they would work interdependently with other class members. Two students said that 

while they were 

excited about taking a distance course, as they would otherwise face a long commute, they had failed to take 

into 

account the additional time the course would require (one of these wrote, "It certainly puts fewer miles 

on one's car 

but I think the time and effort needed are greater than for a regular class"). Seven students volunteered 

some 

strategies for self-management and time management, which will likely be included in the content of 

the final 

product; of these 7, 4 students mentioned particularly the need to set up a comfortable, quiet workspace 

(Figure 6). 

As 

for the technology demands of online coursework, 3 students did write that a basic understanding 

of the particular 

course's 

platform (WebCT, BlackBoard, etc.) would have been helpful for them. Five revealed that they found 

reading from 

a co mputer screen difficult; 2 respondent s mentioned 

the need for an ample supply of paper and printer 

cartridges, indicating 

that they 

were printing 

out much of the course material. In terms of specific technologies, 

one 

student mentione 

d installing plug-ins and three mentioned online chat (Figure 4). In response to the 

quest ion, "What 

is the most difficult 

aspect of online learning?" four students mentioned about discussion forums: responses 

mentioned both the technology requirements as well as the need for competence in composing posti ngs and 

participating in discussions. Three students reported their frustration with problems beyond th eir control, such as 

the 

university's 

network going down, poorly designed features of particular courseware, etc. It seems important to note, 

though, that students for the most part did not perceive deficits in technology skills to be a significant problem. This 

is a striking contrast to the responses of instructors, many of whom spoke at length about their perception that many 

students were sorely lacking in basic technology skills. This may be attributed, in part, to students' lack of awareness 

of magnitude: that is, where one student may have a problem opening a particular file, she or he may not realize that 

the instructor is dealing with that problem with all 20 members of the class at once. 

A number of responses dealt with instructors: thirteen students spoke of unclear instructor expectations, with one 

adding that the lack of non-verbal instructor feedback, such as body language and eye contact, made them uncertain 

as to how to gauge their own performance. Though the question was not specifically asked, it appeared that students, 

for the most part, were not taking the initiative to clarify that information. Five students reported that they had 

concerns with the inaccessibility of online instructors, and 3 noted that instructor skill levels at teaching online were 

critical 

to student success (Figures 4, 5 and 6). 

Interestingly, though most students did report some problems, only 20% of respondents said they would take a 

course on "how to be an online learner" even if it were offered free for 1 hour of academic credit. As will be noted 

later, learner resistance may thus be a problem to the success of the proposed course. 

Advice For New Students 

Student Question 5 (EAC 783 question 6): What advice would you give someone preparing to take an online class 

for the first time? 

1 

96

Time 

Management 

Skills 

12 

10 

8 

6 

4 

2 

0 

11 

5 

4 

Advice for New Students 

3 

2 2 2 

Ti m e Check-in Quiet Space Instructor Technical Supplies Ask for Help Lack 

Management Skill Skills Interaction 

Frequent 

Check-in 

Set up Quiet 

Workspace 

Figure 6: Advice for New Students 

Determine 

Instructor's 

Skill Level 

Before 

Enrolling 

Prepare for 

Technology 

Skills 

Required 

Buy Good 

Printer, 

Paper, 

Toner 

Cartridges 

1 

Ask for Help 

When 

Needed 

Be Aware of 

Lack of 

Interaction 

11 5 4 3 2 2 2 1 

Additional Data 

As the team was concerned about the extent to which the proposed online course might deal with technology 

issues, 

they interviewed two members of the NCSU Learning Technology Services which specializes in assisting instructors 

in putting their courses online. Both were asked, "What do you see as the biggest problems new online learners 

encounter?" They listed several of the s pecifics noted by the instructors, such as saving files, installing plug-ins, and 

configuring 

browsers. The biggest problem, in their estimation, is that many students just seem to have a 

psychological 

block when it comes to new technology: "They don’t try to figure things out, click on the buttons, go 

to th e links, experiment and 

test; I'm especiall y surprised tha t we see so much of this fro 

m graduate students." They 

also mentioned s kills such as using e-mail and attachments, 

but stressed 

that some skills are disciplin 

e-dependent: 

where students in the humanities may need little more than 

e xperience with word processing, 

students in geography 

might have to post a map to the web, and those in business 

fields mi ght need to be proficient in Excel. The 

respondent added that because of this, “The instructor may have to take over the first week of class and help their 

students with these specific skills” rather than try to incorporate everything into a single orientation course for new 

online learners. 

Summary of Findings 

The point on which respondents clearly agreed was the need for online learners to understand the time commitment 

required of an online course and possess or develop strong time management skills. Both students and instructors 

hinted, or wrote directly, that online learning was simply not for everyone and students should make an honest 

assessment of their interests, commitment and abilities before enrolling in an online course. The point on which 

97

espondents differed most was the proficiency level and necessity of technology skills: where instructors found these 

skill deficits to be a large problem, students reported that bigger issues for them were unclear instructor expectations 

and, again, time management concerns. Of the particular technologies specified in both student and instructor 

responses, the most frequent was the use of the discussion forum, both in terms of actual technical competency as 

well as understanding how to compose messages and when to post privately versus publicly. Finally, though most 

students described having some problems during their initiation to online coursework, few indicated that they would 

enroll in a 1-credit-hour 'orientation to online learning course' even if it were offered for academic credit at no cost. 

Implications of Sample size 

Due to the small sample size, 

there is a limitation to generalizing the findings of this study. A low response rate 

raises 

questions about several aspects of the study. First is the notion that those people who returned the instrument 

may be different from those who did not return it. This is called non-response bias (Hawkins, Best, & Cooney, 1992; 

Jobber & O’Reilly, 1996; O’Rourke, 1999). If the response rate had been higher, there would have been a better 

representation of the sample. This lack of representativeness does not ensure biased results, but it does allow for 

greater potential for bias (Bachman, 1999). 

It is possible that respondents were more interested in the study. This could cause them to be more motivated, 

interested or opinionated than those of non-respondents. It has been shown that people who have highly critical or 

favorable opinions are more likely to respond to a survey (O’Rourke, 1999). The low response rate also restricts 

aspects of the analytical techniques used to analyze the data. Overall, it reduces the power of the statistical testing 

(Jobber & O’Reilly, 1996). Low response rates limit the probability that anything other than the strongest differences 

are detected (Hollman & McNamara, 1999). 

As 

a result of the low response rates, the results of this study cannot be generalized to the populations from which the 

samples were drawn. However, the study does provide insights about the perceptions of those people who returned 

the 

instrument. Therefore analyses were run, results were aimed at, and implications directed to only those people 

who responded to the survey. 

Task Analysis 

Duties / Task List 

A job analysis was conducted to obtain a detailed listing of tasks necessary to perform the job of "online learner." 

Team members used their own expertise as online learners to identify key duties and tasks associated with the job, 

acting as Subject Matter Experts (SME). Additionally, feedback from surveys was analyzed to determine other key 

duties and tasks. The tasks identified in this analysis will become a source for identifying the knowledge, skills and 

attitudes an online learner must posses for success in an online course. Tasks are identified beneath the 

corresponding duty. 

Duty 

1.0: Adapt to the online learning environment 

1.1 Prepare for online learning 

1.2 Prepare for online course 

Duty 

2.0: Establish technical resources for online learning 

2.1 Obtain hardware 

resources for online learning 

2.2 Obtain software resources for online learning 

2.3 Access support resources 

for technical resource troubleshooting 

Duty 3.0: Access course web site 

3.1 Log on to university course listings web site 

98

3.2 Access correct class 

3.3 Configure browser for use with class following course recommendations 

3.4 Access support resources for web site troubleshooting 

Duty 4.0: Navigate course web site 

4.1 Navigate course sections via navigation links 

4.2 Navigate subsections within 

main course sections (i.e. things like the course schedule, student home pages) 

4.3 Access support resources for navigation troubleshooting 

Duty 

5.0: Use e-mail to communicate 

5.1 Access e-mail program 

5.2 Send and receive e-mail 

5.3 Manage attachments in e-mail 

5.4 Access support resources for e-mail troubleshooting 

Duty 

6.0: Manage course assignments 

6.1 Submit course assignments on time 

6.2 Manage team assignments 

6.3 Use other software programs applicable to course 

6.4 Access support resources for 

course assignments 

Duty 

7.0: Participate in online discussion 

7.1 Access discussion board 

7.2 Access posts 

7.3 Post an original thread 

7.4 Post responses to a thread 

7.5 Attach a file to a discussion 

post 

7.6 Access support resources for online discussion troubleshooting 

Duty 

8.0: Participate in synchronous chat 

8.1 Access chat 

8.2 Use chat 

8.3 Access support resources for chat function 

troubleshooting 

Duty 

9.0: Complete online quizzes 

9.1 Access online quiz 

9.2 Take online quiz 

9.3 Access support resources for online quiz troubleshooting 

Duty 10.0: Complete online 

assignments 

10.1 Create a student homepage 

10.1 Submit a photograph 

Trainable Tasks 

Once the complete list 

of tasks was determined, it was necessary to select those tasks that require training. Acting as 

Subject-matter Experts 

(SMEs), the team categorized each task according to its criticality, difficulty and frequency of 

99

performance and used the Criticality-Difficulty-Frequency (CDF) model 

(Figure 7) to determine trainability. The 

criticality 

of performance points to the need for selecting tasks for training that are essential to job performance, 

when required, even though the tasks 

may not be performed frequently. Difficulty of task performance refers to the 

time, 

effort and assistance required to achieve proficiency. Frequency of performance is a measure of how often the 

task is performed. Using the straightforward CDF model, all tasks were identified as trainable except tasks 10.1 and 

10.2. These two tasks were 

identified as high difficulty, not critical, and with a low frequency of performance, 

indicating 

that no training was required. 

Recommendations 

For The Online Course 

Figure 7: Cdf (Criticality-Difficulty-Frequency) Model 

As this online course is developed, it will be important to recognize and address student and instructor 

concerns that 

surfaced during the assessment. These concerns should be addressed as part of the course development 

and design or 

by other means to enhance the overall effectiveness of this project. Following are recommendations 

for the 

development of this 

course based on information obtained through analysis. 

This course cannot and is not intended to be a solution to every problem that students and instructors will encounter 

while learning online. One of the first points students and instructors will need to understand is that this 

course is not 

"Computers 101." Learners will be expected to bring to the table basic computer skills and knowledge. 

These basic 

skills should include an understanding of a computer and its use, the ability to keyboard, to power up a computer, 

sign on, use a mouse, click on icons, etc. In other words, this program is not an introduction to computers. 

It will also be critical at the outset to clearly define the course and its objectives to help students 

and instructors 

establish a common ground of expectations. To accomplish this, clear, concise information needs to 

be provided in 

the course description. The course description should list any student prerequisites and outline 

learning objectives the 

course 

hopes to accomplish. To further establish and reinforce common expectations, it is recommended that 

students 

and instructors be provided with a list of competencies that learners would be expected to know upon the 

completion of the course. These competencies would include the ability to: 

100

locate and use support resources for technical troubleshooting. 

access course web sites. 

navigate a course web site including use of navigational links. 

use e-mail. 

open, close, create and send files. 

manage course assignments and meet deadlines. 

participate in online discussions and synchronous chat. 

complete online tests and quizzes as well as complete online assignments. 

If other competencies are then required by an instructor and are not included as part of this course, the instructor 

should be prepared at the outset of a particular course to provide the training or knowledge that will be unique 

to that 

course. 

The course description should clearly establish a realistic understanding of online courses and the knowledge, skills 

and attitudes required by online learning on the part of the student. Online learning has evolved greatly over the last 

few years. In the past, online courses closely resembled "correspondence courses" where assignments were given, 

due dates established and learners responded accordingly. Online learning, with the advancement of technology, can 

now be highly interactive, requiring the learner to participate and interact 

with other learners, teachers and 

facilitators. 

To be a successful online learner in this scenario, learners must possess certain skills and knowledge as 

well an understanding of the time commitment and scheduling requirements 

that may be involved in participating in 

a highly interactive online learning 

experience. Survey results indicated many students had the impression that online 

learning closely resembled the correspondence course of old. While this format does still exist, most online learning 

requires much more 

time, involvement, participation, and interaction than most students anticipate. This gap between 

a student's understanding of online learning 

and its reality will have an impact on student dropout rates as well as an 

impact on the success of students if they do not possess 

the knowledge, skills and attitudes needed to manage this 

typ e of learning experience. Although the purpose of this course 

is not to "sell" online learning, it is again important 

to reiterate that expectations for participants (students and instructors) be defined 

as clearly as possible. 

While the surveys and interviews clearly showed that knowledge and use of technology was a key expectation of 

both the students and instructors, there were other expectations and concerns as well. Technology and technical skills 

are a very important 

component of this online course, but the course will not focus exclusively on these elements. 

The 

team recommends that instructors be made aware that students need and want feedback as well as clearly 

established course requirements and expectations. Some respondents indicated that students were concerned that they 

would not have adequate feedback from instructors online as compared to a traditional classroom situation. Students 

also indicated they were concerned that course requirements and expectations may not be as clear to them as they 

would be face-to-face where they could personally interact and speak to instructors and other students. Instructors 

need to realize that learners are concerned that the feedback they get may be inadequate when compared to a live 

class. Instructors should be aware that learners believe the lack of feedback, eye contact, body language and live 

interaction may negatively impact the learning experience. This need for feedback identified in the survey results 

leads the project team to believe that this course should be facilitated in some part or manner to maximize the 

learning experience. Since the course will represent a learner's first experience with online learning, the project team 

feels it is prudent to support the student as much as possible to better facilitate the transition from face-to-face 

interaction to online interaction. 

To further meet these and other student needs in an online course, it is recommended that instructors be given 

additional training in managing and facilitating an online course. Students also expressed a desire to access help 

when 

needed, whether it was of a technical nature or related to other concerns. As part of this online course students 

should be encouraged to explore and seek answers for themselves. It is recommended that the course provide 

resources for self help and encourage students to explore and seek answers to their questions as part of the learning 

experience before seeking help from an outside source. The findings indicate that students want someone to solve or 

answer their questions before attempting to seek a solution on their own. Procedures and sources for outside help and 

information will be also be provided, but students should be encouraged to seek answers to questions and attempt to 

solve problems on their own. 

101

Students were concerned about the time commitment in an online learning situation. It is therefore recommended that 

a segment on time management be incorporated into the course to assist students in preparing for the time 

commitment that will be needed to successfully take an online course. Also related to the time element was the 

concern among students the course would be "boring" if they already had a mastery of the required skills and 

knowledge needed. As a solution to this concern it is recommended the course be designed so students can test out 

and move as quickly as their individual knowledge and expertise allows. 

Although responses from students and instructors indicated there would be a great need for technical skills and 

knowledge, 

there was a large gap between what students believed their proficiency levels to be and what instructors 

actually experienced in online learning situations. Students assessed their skills as much higher than what instructors 

were actually witnessing. Because they believed their skills were advanced, students indicated they would be 

resistant to taking a course pertaining to online learning. This information further supports the need for a course in 

which students could move at their own pace. Since there appears to be a large gap between actual skill levels and 

perceived skill levels of students, it is recommended that the course be mandatory for all students. This would 

establish a common ground or baseline of operation for students and instructors. Early in the recommendations it was 

stated that this course would provide specific information and skill-building activity to assist students and instructors 

in an online course format. It would be remiss not to recognize that technology is constantly changing and it will be 

important to review and evaluate 

the components of this course on a periodic basis. 

Lastly, the administration of such a course must be taken into consideration. It is recommended that the university 

decide how often and when this course will be offered based on its business plan. Administrators will need to decide 

the time format for the course, whether it should be completed in a few days or over a semester. Although the design 

of this course is for online, information gathered during the assessment indicates there also may be a need for a 

traditional face-to-face version as well. 

Recommendations For Future Research 

This study points to the need for further research in several areas. Instructor skills (or the lack thereof) concerned 

several students: a list of instructor competencies and eventual development of a similar orientation program for 

faculty might be indicated. A study should be undertaken to establish these competencies. 

As both faculty and students saw communication issues as problematic, a study of the ‘rules’ of written 

communication in an academic setting might be appropriate as well. Expectations of both instructors and students 

should be analyzed. 

The delivery of an orientation to online learning program begs for evaluation. Implementation of this course to study 

its effectiveness is essential. Techniques such as pre- and post- studies of the learning 

performance as well as a 

comparison 

of student performance among populations who did attend the orientation versus those who did not. 

Finally, this study should be replicated on a broader scale. A national or international population would add credence 

to the conclusions as well as a global focus on orientation needs. A larger sample size and randomized sampling 

should be used. 

Conclusion 

The 

study elicited several interesting results. Both student and faculty responses indicate the clear need for adequate 

student preparation prior to embarking on an online learning program. Surprising, though, is the disparity between 

the two groups about what such a program should entail. While instructors focused heavily on the need for stronger 

technology skills, the student responses dealt almost entirely with issues of time management, 

personal commitment, 

and 

the need for realistic expectations. Thus, the orientation to online learning program should strive to address the 

expressed needs of both faculty and students. Instructors may also need to be educated about the objectives of the 

program and the content it encompasses, and be prepared to fill in any skill gaps—such as use of a particular tool or 

software program specific 

just to their area of instruction—that students may have upon entering the course. 

Instructors 

also may need to be clearer about their expectations of students and establish protocols for studentinstructor 

communication and feedback. A final issue: while students admitted they needed more preparation, they 

102

paradoxically did not see the need for an orientation program. Thus, the institution offering such a course will need 

to consider making it mandatory (though constructed in such a way that students can ‘test through’ the material) or 

otherwise marketing it to obtain learner commitment. 


Special thanks and acknowledgement goes to Norman Green and Kristen Leonard, members of the original project 

team. 

They made contributions to the basis for this paper. 

References 

Abernathy, D. (1997). A start-up guide to distance learning. Training & Development, December, 39-47. 

Alexander, S. (2001). E-learning developments and experiences. Education + Training, 43(4/5), 240-248. 

Bachman, K. (1999). Diaries: Not so dear. Mediaweek,9(45), 4-5. 

Berge, Z. (ed.) (2001). Sustaining Distance Training. San Francisco: Jossey-Bass. 

Broadbent, B. (2002). ABCs of e-learning: reaping the benefits and avoiding the pitfalls. San Francisco: Jossey – 

Bass/Pfeiffer. 

Charp, S. (1994). Viewpoint. The On-line chronicle of distance 

education and communication, 7(2). 

Driscoll, 

M. (2002). Web-based training: creating e-learning experiences. San Francisco: Jossey-Bass/Pfeiffer. 

Hardy, 

D. & Boaz, M.. ( 1997). Learner development: beyond the technology. In T.E. Cyrs (ed.), Teaching and 

learning at a distance: what it takes to effectively design, deliver and evaluate programs. San Francisco: Jossey-Bass 

Publishers. 

awkins, D.I., Best, R.J. & Coney, K.A. (1992). Consumer behavior (5 th H 

Ed.). Homewood, IL: Irwin. 

Hollman, 

C.M. 

& McNamara, J.R. (1999). Considerations in the use of active and passive parental consent 

procedures. The Journal of Psychology, 133(2), 141-156. 

Horvath, A. & Teles, L. (1999). Novice users reactions to a web-enriched classroom. Virtual University 

Journal, 

2(2), 

44-49. 

Jobber, 

D. & O-Reilly, D. (1996). Industrial mail surveys: techniques for inducing response. Marketing 

Intelligence& Planning, 14(1), 29-34. 

Mason, R. (2001). Time is the New Distance? Inaugural lecture, The Open University, Milton Keynes. Retrieved 

March 11, 2002, from http://kmi.open.ac.uk/stadium/live/berrill/robin_mason.html. 

Moore, M. (1991). Distance education theory. The American Journal of Distance Education, 5(3), 1-6. 

O’Rourke, T.W. (1999). The importance of an adequate survey response rate and ways to improve it. American 

Journal 

of Health Studies (15)2, 107-109. 

Schlosser, C. & Anderson, M. (1994). Distance education: review of the literature. Washington, DC: Association 

for Educational 

Communications and Technology. 

Sherry, L. (1996). Issues in Distance Learning. International Journal of Educational Telecommunications, 

1(4), 337- 

365. 

Willis, B. (1992). Strategies for teaching at a distance. (ERIC 

Document Reproduction Service No. ED351008). 

103

Table 1: Online Resources 

Table 1 lists online resources that were accessed and examined as extant data in the analysis process. 

Resource 

Germanna Community College – Time 

Management for College Students 

Web Address 

http://gcclearn.gcc.cc.va.us/adams/time/ 

Fastfax – Effective Time Planning http://www.learningcommons.uoguelph.ca/learning/fastfax/planning. 

Strategies htm 

University of Guelph – Learning Time http://www.webshops.uoguelph.ca/learningtime/ 

Harcourt College – Time Management http://www.hbcollege.com/management/students/bktimemg.htm 

UNC – CH – Blackboard Course http://blackboard.unc.edu 

NAU – Online Learner's Guide http://jan.ucc.nau.edu/~d-ctel/OLG/Welcome.php 

Central 

Piedmont Community College – http://cww.cpcc.cc.nc.us/readiness/index.asp 

Distance Learning Readiness 

Minnesota Virtual University – Online http://www.mnvu.org/Frame?pg=3035 

Learner 

Technology and Telecommunications for http://www.k12.hi.us/~tethree/00-01/content/onlinelrnr.htm 

Teachers – Being an Online Learner 

WebCT – Student Orientation Center http://www.webct.com/oriented/viewpage? 

Illinois Online Network – What Makes a 

Successful Online Student? 

Illinois Online Network – Self-Evaluation 

for Potential Online Students 

http://www.ion.illinois.edu/IONresources/onlinelearning/StudentProf 

ile.html 

http://www.ion.illinois.edu/IONresources/onlinelearning/selfEval.ht 

ml 

Illinois Online Network – Tips for Online 

Success 

http://www.ion.illinois.edu/IONresources/onlinelearning/tips.html 

NCSU – Distance Education http://distance.ncsu.edu/green/de4me.htm 

John's Hopkins – Distance Learning Course http://distance.jhsph.edu/oll 

Table 1: Online Resources 

104

Appendix A –Student Questionnaires 

Student Questionnaire, EAC783 

Posted in online discussion forum 

Students in EAC 783 Fall 2002 

Option to post responses to board or reply to surveyors via email 

1. If you could have learned something about online 

learning prior to beginning your first online course, what 

would have been helpful? (E.g., sending email, chat, downloading, how to post discussion messages, 

instructor 

expectations, time commitment.) 

2. If a free 1-credit-hour course had been offered 

online to better prepare you for your first online course would 

you have taken it? Why or why 

not? 

3. What were your original assumptions about online learning and have they changed? Why or why 

not? 

4. Now that you are participating in an online course, what do you think is the most difficult aspect of online 

learning? 

5. Would you recommend an online course to others? Why or why not? 

6. What 

advice would you give to someone preparing to take his/her first online course? 

Student Questionnaire, EAC784 

Sent via email 

Students in EAC 784 Fall 2002 

Responses via reply email (not anonymous) 

1. If you could have learned something about online 

learning prior to beginning your first online course, what 

would have been helpful? (E.g., sending e-mail, chat, downloading, how to post discussion messages, instructor 


2. If a free 1-credit-hour course had been offered online to better prepare you for your first online course would 

you have taken it? Why or 

why not? 

3. What was your original assumption(s) about online learning and has it changed? 

Why or why not? 

4. 

Now that you are participating in an online course, what do you think is the most difficult aspect of online 


5. Which would you recommend to others: online learning or standard instructor-led classroom training? 

Student Questionnaire, LTS 

Posted by NCSU Learning Technology Services for all students enrolled in online classes 

Fall 2002 

Respondents could elect to submit responses anonymously 

1. If you could have learned something about online learning prior to beginning your first online course, what 

would have been helpful? (E.g., sending e-mail, chat, downloading, how to post discussion messages, instructor 


2. If a free 1-credit-hour course had been offered online to better prepare you for your first online course would 

you have taken it? Why or why not? 

3. What was your original assumption(s) about online learning and has it changed? Why or why not? 




105

Student Questionnaire, Corporate Employees enrolled at University of Phoenix 

Data collected through email communication 

with employees from a local corporation who were participating in 


learning through the University of Phoenix. 

1. If you could have learned 

something about online learning prior to beginning an online course, what would 

have been helpful? (E.g., Sending email, Chat, Downloading, 

How to post discussion messages, Instructor 

expectations, Time commitment). 

2. If a free 1 credit hour course were offered online to help you learn more about how to be an Online Student, 

would you take it? 

3. What was your original assumption 

about online learning and has it changed? 




Appendix B – Instructor Questionnaire 

Distributed to instructors of EAC 783 (Advanced 

Instructional Design) and 784 (Needs Assessment and Analysis) 

Posted online to all instructors of online courses 

at NCSU Fall 2002 

(respondents 

had option to reply anonymously) 

Distributed to several 

instructors of online courses at UNC-Chapel Hill 

1. If a student had already completed a 1 credit 

hour online course preparing them for Online Learning, what 

would you expect them to know upon entering your class? (Technology, online participation, realistic 

expectations of workload, etc.) 

2. What are the top problem areas that online students have encountered in an online class (Technology, Design, 

User-Knowledge, etc.) 

3. What problem areas are not within the student’s control? 

4. What frequently asked questions do you 

receive that are not within your expertise? 

5. What is the average dropout rate for an online course? Of the students who dropped, what are their common 

reasons for dropping? 

6. How many students in an online class is too many? 

7. Typically, how many hours per week does one of your students devote to an online class? 

106

Salmon, D., & Jones, M. (2004). Higher education staff experiences of using web-based learning technologies. Educational 

Technology & Society, 7 (1), 107-114. 

Higher education staff experiences of using web-based learning technologies 

Abstract 

Debra Salmon and Mathew Jones 

Faculty of Health and Social Care 

University of the West of England 

Bristol, BS16 1DD 

United Kingdom 

Tel: +44 117 328 8468 / 8769 

Fax: +44 117 328 8437 

Debra.Salmon@uwe.ac.uk 

Matthew.Jones@uwe.ac.uk 

Given the drive in higher education institutions to employ web-based learning (WBL) technologies in their 

curricula, this article sets out to address the question of how staff experience the incorporation of such 

technologies into their educational practice. The study focuses on an initiative involving four institutions in 

South and West England that aimed to encourage the strategic development of WBL resources in health and 

welfare professional education programmes. 

Thirty-three higher educational staff from a range of organisational locations took part in a qualitative 

process study. The findings suggest that while staff were enthusiastic about their engagement with WBL 

they experienced problems embedding their project work within their organisations, managing their time 

and obtaining institutional recognition for their work. Such findings represent a challenge to commentaries 

that emphasise “technological illiteracy” or “technophobia” amongst staff as barriers to WBL 

implementation. The study favours an analysis that emphasises how WBL initiatives are incorporated into 

existing higher education managerial, decision-making and reward structures. 

Keywords 

Higher education, Staff, Web-based learning, Organisational research 

Introduction 

Over the past decade higher education institutions (HEIs) have increasingly become involved in the use of webbased 

learning (WBL) (DiPiro, 1999; Joyes, 2000; McNaught & Kennedy, 2000; Spratt et al, 2000). Whilst this 

engagement has been uneven across the UK higher education sector, most universities have begun to explore the 

role of WBL in redefining boundaries between conventional and distance-based programmes and the prospects 

for entering into WBL-based consortia with other HEIs and educational organisations. With reference to the use 

of new technologies in distance learning in higher education, Tearle et al state “it is no longer possible to opt 

out” (1999:14). As WBL becomes a mainstream issue for HEIs, this paper sets out to address the question of 

how staff experience the incorporation of such technologies into their educational practice. The experiences of 

staff working in health and welfare professional education form the basis of our study. 

In the UK, the need for academics and other staff to engage with new learning technologies in higher education 

was most visibly propounded in the Dearing Report (Dearing, 1997). According to the report, the combination of 

continued HE expansion with ongoing financial constraint requires a major re-think of how education is 

delivered. New learning technologies offer a solution to this problem and have, in addition, been claimed to 

transform and enhance the student learning experience (Department of Health, 1999; Edwards & Clear, 2001; 

Peach, 1999; Richardson et al, 1999; Smith & Hardaker, 2000; Tearle et al, 1999; Thomas et al, 1998). 

The adoption of new learning technologies implies significant changes in the working environment of lecturers, 

managers and other HE staff. The Dearing Report stated that “many academics have had no training and little 

experience in the use of communications and information technology as an educational tool” (1997: 36). A 

number of commentators have identified such ‘technological illiteracy’ as the big barrier to the take up of WBL 

in HE settings (Joyes, 2000; Rossiter & Bagdon, 1999). And, in pursuit of this theme, others have focused on a 

deep seated ‘technophobia’ amongst academics who are apprehensive of radical, unproven innovations and 

resistant to changes that might undermine their professional status (Johnston, 1997; Spratt et al, 2000; Thomas et 

al, 1998). However, as Vermeer’s review points out, such commentary suffers from anecdotal claims and “the 






107

enthusiasm of the recently converted” (Vermeer, 2000:329): consequently some writers have had a tendency to 

over-focus on the knowledge and attitude deficits of staff. 

Other literature points to a more complex picture: staff experiences of WBL may have had more to do with the 

wider organisational context than the characteristics of the technologies themselves. Ward and Newlands’s study 

found that for many lecturers “the problem with technology driven innovations is that they can consume 

prodigious amounts of time and money to little educational effect” (Ward & Newlands, 1998:171). McNaught 

and Kennedy’s study (2000) of the development of WBL course materials found the process time consuming and 

demanding of high levels of technical support. Cravener’s review (1999) also identified several reports of 

increased staff workload under such conditions. 

There is little direct literature on how staff receive recognition for these kinds of activities in HE, however 

studies of distance learning have drawn attention to weak institutional commitment within mainstream HEIs. 

Wolcott found that the HE reward systems were not accommodating to staff undertaking innovative educational 

projects: “institutional rewards are not in sync with alternative forms of delivery" (1997: 17). For participants, 

such projects actually involved a high degree of ‘career risk’ because they were drawn away from more 

conventionally recognised academic pursuits. In a context of collegiate academic practice, Tearle et al (1999) 

and Browning and Williams (1997) similarly identified an absence of career development benefits for staff who 

developed information technology (IT)-based learning resources. 

Project description 

The Interactive Teaching and Learning (INTaL) initiative was funded by the National Health Service Executive 

(South and West). It aimed to encourage the strategic development, implementation and evaluation of interactive 

teaching and learning material for use in educational programmes for professional practitioners in the health and 

welfare professions. Collaboration in the project spanned four HEIs across the South and West of England. 

The initiative had several objectives including: staff development sessions on the creation and use of interactive 

teaching and learning material; the creation of an WBL interest group; and the development of CD-ROM based 

learning materials. The final objective was to encourage increased use of interactive teaching and learning within 

at least one educational programme in each institution. This last objective led to the development of a set of 

projects on the following issues: partnership working for public health, hospital discharge planning, teenage 

pregnancy, mental health, diabetes in older people, visual impairment and organisation resource management. 

This article reports on the evaluation of staff involvement in these projects. 

Research Aims and Methods 

In contrast to other studies that have focused on evaluating the impact and outcomes of WBL initiatives in HE 

settings (e.g. Cartwright 2000, Kozlowski 2002), we took the focus of our study to be the experiential accounts 

of project participants. The research aimed to examine different project staff experiences according to their 

location within the participating HE organisations. Following existing literature, there was speculation that the 

use of WBL may pose challenges for lecturers accustomed to working in conventional HE settings. Staff at all 

levels of the project development were requested to participate in the study. Anonymity was assured for all 

those taking part and agreement was given by all those contacted. A purposive sample of key players were 

identified and included INTaL steering group members; project leaders; project members; line managers of 

project leaders and members and the INTaL manager and officer (see Table 1). Substitute participants were 

sought in cases where participants in the evaluation exited the project teams. 

Drawing upon methodologies in qualitative process evaluation (Maxwell, 1996), the fieldwork took place in two 

stages: at the beginning and at the end of the project. The main method of data collection was recorded face-toface 

interview. Where there were difficulties accessing respondents, telephone interviews and semi-structured 

questionnaires were used. Subjects covered in the initial interview included: motivation for engagement together 

with the perceived benefits of web-based projects, experiences of involvement, support mechanisms, and the 

perceived relationship between the project and broad organisational objectives. 

In the second stage of data collection participants were asked to reflect upon the degree to which their 

expectations had been consistent with the overall project aims. As with the initial interviews, staff were asked to 

comment on factors helping or hindering their involvement; levels and types of support received; and the ways in 

108

which the project would be used within their organisations. Interviews with managers took a different format: 

they were asked to comment on staff support, resource allocation, the benefits for staff involvement and the 

possibilities for future development. 

Table 1: Role and Institution of Respondents 

Role in the case study Numbers of 

participants 

interviewed 

Steering group member 9 

Project leader 8 

Project member 10 

Line manager of project leaders and members 4 

INTaL manager and officer 2 

Total 33 

Institution 

Institutions A, B, C, D 

Institutions A, B, C 

Institutions A, B 

Institutions A, B, C 

Institution A 

Qualitative data from both the interviews and questionnaires were categorised and analysed for emergent themes 

using the thematic approach set out by Strauss and Corbin (1998). The organisation of the qualitative data was 

supported through the use of the data analysis package, NUD*IST version 4. 


Experiences and initial outlook 

All staff involved in this project had at least five years experience of working in higher education. All the project 

leaders and members described themselves as at least ‘fairly experienced’ in IT. The level of enthusiasm 

articulated about the possibilities of developing new approaches to learning was noteworthy. Common themes 

for becoming involved in the project were categorised as: the desire to extend a role in IT; personal and 

professional interests in new IT; and the opportunity to extend existing or create new educational resources. 

The reasons described above illustrate how decisions to become involved were not directed by managers. Staff 

largely experienced the project as a bottom-up initiative in which they could choose to become involved. They 

also described feeling a strong sense of academic responsibility to the organisation that acted as a significant 

motivational factor. The desire to extend and develop collaborative working across different professional groups 

and across organisations was also perceived as an important reason for involvement: 

Through working with the other institutions you are going to get different perspectives; different approaches to 

doing things with the way that material is put together or thinking about assessment. We might try and go along 

with [my University’s] frame in our minds and they might come with a different perspective that might help 

develop things. [CSL1] 

I think potentially [collaboration] is an excellent idea because it seems daft to keep re-inventing the wheel […]. 

But in practice every institution has it’s own different courses and I’m not sure how well something like this 

would actually be useable by different institutions. I would very much like to see is more collaboration, but then 

you have the politics of who owns the course, who pays for it and what happens when people sign up to it. 

People are in competition - all the different institutions want students and are desperate for that market. [CSL3] 

Participants held a set of assumptions about the nature of the WBL that included the view that this type of 

educational approach was inherently progressive: 

It’s seductive. So this may well be seen as “Brilliant, this is what we should be doing - Oh yes” ! [CSL6] 

109

The first thing is, it’s different. Students have a traditional view of books, libraries, sitting down reading, 

lectures and listening. All these things are not interactive; they are very much either being fed material or just 

turning over pages. So it’s different. [TS1] 

I feel the sooner we get to some sort of technological age the better, I can’t be dealing with the “History of 

Nursing and Florence Nightingale”. I really don’t give a toss; she was probably a syphilitic ridden old bag. It’s 

just the way forward! [CSL7] 

For some participants it was felt that the novelty of a new approach might inspire collaboration between different 

health and welfare professionals and across organisations. It was also perceived as a route for saving institutional 

costs in the context of the mass delivery of higher education: 

We have reached the situation where we have too many students to be able to [teach] in any other way but as an 

electronic community. [CSL5] 

Where you have got more students in a square area and you can actually do more… the need for classrooms is 

less, so you save on University overheads. If we could have students sitting in their home, running a learning 

resource web at a time that suits them, they can tune in to their own biorhythms … and all the benefit will come 

from [working in] their own time frame. [CSM7] 

[An attraction is] the fact that you can appeal to a much larger consumer base than you otherwise could. There 

is ready access to users and also, because you are appealing to a fairly wide Internet user community, you could 

bring in quite a lot of feedback which enables you to validate and upgrade the material relatively easily. [CSM2] 

Staff were also conscious of the claims and objectives of these sort of initiatives. When asked to anticipate the 

impact on students of using the newly developed web-based projects participants identified several key issues. 

These included the immediacy of communication with students, promoting student interaction, increasing access 

to learning resources, flexibility of access in time and space and the promotion of independent learning. 

The process of involvement 

Participants were asked about their experiences of working in teams and of collaborative working with 

colleagues in the development of WBL materials. A dominant theme from the interviews concerned the 

difficulties getting groups to become established, to convene and to exchange ideas, knowledge and skills. This 

was in part a consequence of competing priorities and lack of clarity regarding the level and nature of 

involvement. Consequently project leaders and technical support staff all felt a high level of responsibility to 

build and motivate teams. 

A specific issue participants raised was around project objectives. Tensions emerged between whether the 

projects were to be judged in terms of process or outcome outputs. Lack of clarity led to feelings of anxiety, 

frustration, confusion and stress amongst participants: 

Initially I thought I understood, then the project evolved from what it was in the beginning. From that point of 

view I go in and out of understanding what is required of me, and then being totally confused about what is 

required of me. [CSL1] 

One of the problems with me is not knowing where the project is going to be used. That’s the real problem for 

me because I am very pragmatic. I like to know, who I am designing it for, how it is going to be used and 

without being very clear, I am not entirely certain where I am going with it. [CSL30] 

All staff (n=18) directly contributing subject expertise reported difficulties translating teaching materials that 

evolved from individual scholarly study into collaboratively produced WBL materials. As the last extract 

illustrates, all staff reported a high degree of autonomy and self-management in their delivery of existing 

programmes. There were concerns that conversion to WBL delivery implied a move away from a collegiate 

cottage industry towards a corporate, standardised and centrally managed curriculum. Thus the relationship 

between their WBL project work and the priorities of the organisation was brought into sharp focus. Six 

participants stated that it was difficult to see how the products of their work fitted into other curricula activities. 

The work fell outside the routine priorities of the organisation and as such was not interpreted through 

recognised workload frameworks: 

110

We thought our work would fit in the new curriculum, but having spoken to [a senior manager] it would seem 

that this idea is not being taken forward. There is no expectation that we will be using web technology for those 

modules. So now I’m not sure where it is going to fit. [CSL3] 

This was an issue raised by staff from all the participating organisations. 

The majority of participants stated that questions of time allocation and relief from other work had remained 

unresolved throughout their involvement with the project. Three respondents stated that outline agreements with 

managers did not correspond to actual involvement. In addition, while participants did have managerial and 

technical support, they found that technological problems consumed frustrating amounts of the time and energy: 

I think time is incredibly difficult really. My line manager has been brilliant: very good in looking at time 

schedules. But it is so difficult to predict how much time it is going to take. [CSL4] 

In a sense, I manage myself because I am doing research there is no-one who can take over that particular part 

of my work, any increased time allocation we could devote, this would really be quite notional because there are 

still other things I have to do. [CSM2] 

All the managers interviewed expressed an enthusiasm for staff involvement in developing WBL materials. They 

saw it as an important and exciting opportunity for staff that was consistent with the long-term strategic vision of 

their university. However, there were reservations about the lack of guidance about how to recognise WBL 

development in staff workload arrangements. Questions were raised about how to generate time relief, 

effectively support staff and disseminate the project findings more widely. In particular, managers were 

confused about whether or not there were funds to ‘buy-out’ staff time. One manager said: 

What concerns me is that as I had never been asked to allocate time, I now understand I should have been giving 

them half a day a week, and they never came and asked me. This had repercussions in terms of staff struggling 

to meet the deadlines set by the initiative. Communication appeared to have broken down between the project 

and line managers. [MGR1] 

Review of the project 

The final findings section represents staff experiences as they reviewed the project, in particular the factors that 

had supported and hindered their involvement. For five of the participants the collaborative elements of the 

project had failed to materialise. However, some respondents emphasised links that had been forged, in 

particular, with colleagues in clinical practice. It was also identified that in complex organisations, a lack of 

collaboration between initiatives themselves can lead to a failure to embed projects in organisational cultures. 

Managers in the following extracts commented upon this peripheral location: 

The problem with what happens here, when you have major projects developing. It is like lifts on the outside of 

a building with no connection in between. You just get out on separate floors and there is no connecting 

doors...The added problem was that INTaL didn’t have a high enough profile...no it didn’t have any profile. 

[MGR3] 

One of the difficulties about the project has been that it exists on the periphery of everyone’s vision. It exists on 

the very outside of everyone’s workloads and there never could be, and there never was written into the project, 

it being integrated into the curricula in any shape or form. [MG2] 

Lack of knowledge of IT created a hindrance for a minority (4 out of 18) staff involved in the development in the 

WBL projects. This was not due to lack of IT advice, with regard to the technical aspects of WBL design and 

delivery, but rather an opportunity to develop a more thorough grounding in the skills required to develop WBL 

or the opportunity to make a more specialist contribution as a member of a team. 

Participants identified a number of ingredients for success for any subsequent WBL projects. These could be 

summarised in terms of clear objectives, appropriate time allocation and project deadlines, and clearly defined 

roles and responsibilities. Whilst specialised career pathways were perceived to be appropriate for driving the 

WBL agenda at a faculty level, participants also highlighted the need to recognise and reward mainstream staff 

who develop WBL as one aspect of their work. Regular face to face contact with other team members and 

technical support was also perceived as helpful as was specialist advice around issues such as web site design 

111

and presentation. Overall participants highlighted the need for a more task orientated and structured process that 

implied a less ad hoc and self-managed approach. This was in contrast to the relatively autonomous working 

practices to which the participants were accustomed. 

Discussion 

As the use of WBL becomes an issue for mainstream academic and associated staff in HEIs, it is important to 

understand the barriers and opportunities to development in this field. This study found that participants felt selfmotivated 

and enthusiastic in their engagement with the use of WBL technologies at the outset. This outlook not 

only inspired them to join the project but also sustained them when problems arose. Participants wanted to both 

protect academic standards and enhance opportunities for student learning. Generally it was perceived that this 

could be achieved through working collaboratively, pooling knowledge and skills, spreading good practice, and 

developing new networks both within their own institutions and with service providers. As the project 

progressed, participants experienced problems embedding their project work within their organisations, 

managing their time and obtaining institutional recognition for their work. Previous studies have tended to focus 

on specific technological characteristics to account for HE staff and organisational experiences (e.g. Joyes, 2000; 

Spratt et al, 2000). However the view that ‘technological illiteracy’ and ‘technophobia’, constitute key barriers 

(ibid.) to the use of learning technologies in higher education was not supported in this study. 

As McNaught and Kennedy’s (2000) study found, the web-based programmes under development in this project 

required greater time and resource commitments in comparison to conventional courses. The initiative also drew 

a number of HEIs into complex collaborative arrangements. These features presented difficulties in the 

organisational HE settings represented this study. Furthermore, the development of web-based programmes 

implied a team project approach with a clear set of project objectives, definite time schedule and directive 

communication. To some extent, greater attention to planning and project management may have addressed 

these issues. However the findings draw attention to the interaction between WBL development and specific 

characteristics higher educational academic decision making structures. Staff experienced difficulties reorientating 

their collegiate and relatively self-directed working practices to meet the demands of a WBL 

initiative. For similar reasons, Cravener’s review (2002) indicates that HEIs often struggle to hold together 

project teams for online course development. One challenge, therefore, for the mainstreaming of WBL lies in 

how to resolve the tensions between academic self-management and organisational project management in HE. 

Given that web-based technologies have been in use for over a decade, it was interesting to find that staff 

perceived real difficulties translating their existing teaching programmes into web-based delivery. Participants 

felt uncertain about how well they could justify their use of time in the initiative. As previous studies have 

demonstrated (e.g. Tearle et al, 1999; Wolcott, 1997), participants in this study were unclear about how their 

activity was being recognised within existing workload arrangements. This was partly a result of fitting their 

project work into broader curricula activities. It was significant that staff line-managers also reported difficulties 

incorporating the projects into organisational practices. In a field imbued with assumptions of the inherent 

technological progress, IT learning projects have often had unpromising track records. The study supports some 

recent literature concerned with how organisations embed IT learning initiatives within existing the managerial, 

decision-making and reward structures of HE (Carswell et al, 2000; Edwards & Clear, 2001; Huff, 2000; 

Saunders & Weible, 1999; Smith & Hardaker 2000). 

Conclusions 

At present HEIs are employing a number of strategies for promoting the use of WBL. Much emphasis has been 

placed upon staff knowledge, skills and attitudes towards the technological aspects of the new media. Working 

from the experiences of staff, this study indicates that the key issues arising from the mainstream use of WBL 

relate to academic working practices and organisational management. The specific technical characteristics of 

the learning technologies were only the backdrop to the social relationship aspects of the project. For HE 

academic staff there was an issue of how they adapted established modes of academic practice to a more teambased 

project orientation. Meanwhile HE managers needed to promote relationships between staff with a mix of 

academic, educational, technical and design expertise. Whilst HEIs are developing career structures for staff 

pursuing specialised work in WBL developments (Cravener, 2000; Hughes & Daykin, 2002), HE managers also 

need to develop clear mechanisms for supporting non-specialist staff who are involved in WBL developments as 

part of their role. Such support is particularly needed in terms of time allocation, role recognition and reward 

112

structures (see also Torrisi & Davis, 2000). Ultimately, the way forward for WBL involves recognising new 

forms of HE working relationships. 

The study setting examined here is limited to the HE health and social care academic environment in one region 

of the UK. It must be recognised that the findings discussed here may not be generaliseable to other HE contexts. 

Nevertheless, qualitative studies are important for exploring lived experiences with educational technologies and 

the incorporation of new technologies into existing organisational practices. There is a need for further research 

on the perspectives of staff who choose not to adopt WBL in their programmes as well as those who try WBL 

and subsequently revert to more traditional modes of delivery. There remains scope for organisational analysis of 

the interaction between ‘technological imperatives’, HE managerial agendas and academic professional 

responses in the field of new learning technologies. It is suggested that further empirical study on the impact of 

new learning technologies upon staff experiences, academic practices and managerial structures in HE would 

inform critical debate in this field. 


We would like to express our thanks to all the staff who participated in this study, particularly those who gave 

their time to lengthy interviews and questionnaires. We would also thank Gill Hek and Paul Gilbert of the 

steering group for their support throughout the project and Leigh Taylor for her excellent administrative support. 

References 

Browning, P. & Williams, J. (1997). Using the internet in teaching and learning: A UK perspective, Computers 

And Geosciences, 23 (5), 549-557. 

Carswell, L., Thomas, P., Petre, M., Price, B. & Richards, M. (2000). Distance education via the internet: The 

student experience, British Journal Of Educational Technology, 31(1), 29-46. 

Cravener, P. (1999). Faculty experiences with proving online courses: Thorns among the roses, Computers in 

Nursing, 17 (1), 42-47. 

Dearing, R. (1997). Higher education in the learning society: Report of the National Committee Of Inquiry Into 

Higher Education, London: HMSO. 

Department Of Health, (1999). Making a difference. Strengthening nursing, midwifery and health visitor 

education, London: Department Of Health. 

DiPiro, J.T. (1999). The Virtual University –higher education ‘on-line’, American Journal Of Pharmaceutical 

Education, 63 (2), 170-174. 

Edwards, M. & Clear, F. (2001). Supporting the collaborative learning of practical skills with computermediated 

communications technology. Educational Technology & Society, 4 (1), 80-92 

Huff, W.D. (2000). Colleges and universities: Survival in the information Aage, Computers And Geosciences, 26 

(6), 635-640. 

Hughes, M, & Daykin, N. (2002). Towards constructivism: Investigating students’ perceptions and learning as a 

result of using an online environment. Innovations in Education and Teaching International, 39 (3), 217-224. 

Johnston, R. (1997). Distance learning: Medium or message? Journal Of Further And Higher Education, 21(1), 

107-122. 

Joyes, G. (2000). An evaluation model for supporting higher education lecturers in the integration of new 

learning technologies, Educational Technology & Society, 3(4), 1-14. 

Kozlowski D. (2002). Using online learning in a traditional face-to-face environment, Computers in Nursing, 20 

(1), 23-30. 

113

Maxwell, J. (1996). Qualitative research design: An interactive approach. London: Sage. 

Mcnaught, C. & Kennedy, P. (2000). Staff development at RMIT: Bottom up work serviced by top down 

investment and policy, In: Squires, D. Et Al (2000) The changing face of learning technology, Cardiff: 

University Of Wales. 

Peach, L. (1999). Fitness for practice: The UKCC Commission for nursing & midwifery London: UKCC. 

Richardson, J. T. E., Morgan, A. & Woodley, A. (1999). Approaches to studying in distance education, Higher 

Education, 37 (1), 23-55. 

Rossiter, D. & Bagdon, K. (1999). Embedding the acquisition of technological literacy: A case study, 


Saunders, G. & Weible, R. (1999). Electronic courses: Old wine in new bottles? Internet Research – Electronic 

Networking Applications And Policy, 9 (5), 339-347. 

Smith, D. & Hardaker, G. (2000). E-learning innovation through the implementation of an internet supported 

learning environment, Educational Technology & Society, 3 (3), 422-432. 

Spratt, C., Palmer, S., & Coldwell, J. (2000). Using technologies in teaching: An initiative in academic staff 

development, Educational Technology & Society, 3 (3), 455-461. 

Strauss, A. & Corbin, J. (1998). Grounded theory in practice, London: Sage. 

Tearle, P., Dillon, P., & Davies, N. (1999). Use of information technology by English university teachers. 

Developments and trends at the time of the National Inquiry Into Higher Education, Journal Of Further And 

Higher Education, 23(1), 5-15. 

Thomas, P., Carswell, L., Price, B. & Petre, M. (1998). A holistic approach to supporting distance learning using 

the Internet: Transformation, not translation, British Journal Of Educational Technology, 29(2), 149-161. 

Torrisi, G. & Davis, G (2000). Online learning as a catalyst for reshaping practice – the experiences of some 

academics developing online learning materials. The International Journal for Academic Development 5 (2), 

166-176. 

Vermeer, R. (2000). Review of Internet based learning: An introduction and framework for higher education and 

business. Computers & Education, 35(4), 327-333. 

Ward, M. & Newlands, D. (1998). Use of the web in undergraduate teaching, Computers & Education, 31(2), 

171-184. 

Wolcott, L. L. (1997). Tenure, promotion and distance education: Examining the culture of faculty rewards 

American Journal Of Distance Education, 11(2), 3-18. 

114

Tisdell, E. J., Strohschen, G. I. E., Carver, M. L., Corrigan, P., Nash, J., Nelson, M., Royer, M., Strom-Mackey, R., & 

O’Connor, M. (2004). Cohort Learning Online in Graduate Higher Education: Constructing Knowledge in Cyber 

Community. Educational Technology & Society, 7 (1), 115-127. 

Cohort Learning Online in Graduate Higher Education: Constructing 

Knowledge in Cyber Community 

Elizabeth J. Tisdell 

Associate Professor of Adult Education 

Penn State University-Harrisburg 

777 W. Harrisburg Pike 

Middletown, PA 17057-4898, USA 

Tel: +1 717 948-6640 

Fax: +1 717 948-6064 

ejt11@psu.edu 

Gabriele I. E. Strohschen 

Assistant Professor and Director of Graduate Programs of 

Applied Professional Studies and Applied Technology 

DePaul University 

25 E. Jackson Blvd. 

Chicago, IL 60604, USA 

Tel: +1 312 362-5122 

Fax: +1 312 362-8809 

gstrohsc@depaul.edu 

Mary Lynn Carver, Pam Corrigan, Janet Nash, Mary Nelson, Mike Royer, 

Robin Strom-Mackey, Marguerite O’Connor 

c/o Department of Adult and Continuing Education 

National-Louis University 

122 S. Michigan 

Chicago, IL 60603, USA 

Tel: +1 312 621-9650 

Fax: +1 312 261-3057 

Abstract 

This paper discusses a qualitative participatory action research study, which examined the nature of the 

cohort learning experience in an online master’s program, from both faculty and student perspectives. After 

describing this online master’s program in adult education designed from a social constructivist theoretical 

frame, this paper discusses two primary areas of findings related to cohort learning. First, were those 

related to the ongoing negotiation of the learning process: the importance of an opening residential; a 

consistent but flexible cohort structure; and building ongoing relationships. Second, were those related to 

the ongoing construction of knowledge: the role of team-teaching and the cohort model in transformative 

learning; the application of theory to real life practice, and the value of group support and collaboration in 

conducting research and constructing knowledge. Implications for practice are discussed. 

Key words: 

Cohort learning, online learning, online adult degree programs, residential learning, social constructivism 

In the last decade, there has been much discussion in higher and adult education circles on distance education, 

Internet-based and Internet-enhanced learning, and online degree programs. Most institutions of higher education 

now offer at least some classes online, and many offer entire degree programs. While many have discussed the 

plusses and minuses of online education, and considered what it offers adult learners, there has been little 

discussion of online education in Internet-based cohort programs, particularly from the students’ perspectives. 

Cohort-based degree programs are those programs where the same group of students begins a degree program 

together, takes the same sequence of courses, and, assuming they successfully complete each course, graduate 

together. While there has been some discussion of cohort based adult degree programs in the past 20 years, there 

has been very little consideration of the online cohort learning experience, Thus, the purpose of this paper is twofold: 

(1) to discuss the results of a participatory action research project which specifically examined the nature 

of the cohort learning experience in an online master’s program that began with a residential component, from 

both faculty and student perspectives; and (2) to consider the implications for the ongoing development of and 

academically sound degree programs in adult education and related areas. 






115

Related Literature 

There is no question that online degree programs meet many of the needs of adult learners. Nevertheless, 

learners generally participate in distance education programs not so much because they prefer them to oncampus 

face-to-face (FTF) instruction, but because computer mediated communication (CMC) and instruction 

provide a way to reach their personal goals despite constraining personal circumstances (Green, 1999; Northrup, 

2002). As Sherron and Boettcher (1997) suggest, the main reasons for the proliferation of degree programs 

online are the availability of communication through computing technologies; the need for workers to acquire 

new skills without interrupting their working lives for extended periods of time in this information age; and the 

need to reduce the administrative cost of education. 

In light of the proliferation of online degree programs, there has been much literature in the past ten years on the 

reasons for the development of online learning, suggested ways of teaching online, and research studies on the 

nature and impact of online learning and CMC in higher and adult education settings (Dede, 1996; Greene, 1999; 

Schrumm, 1998). Some of the skepticism of the very early years of online higher education classes and degree 

programs has given way to increased discussion of how to teach more successfully online, along with the 

recognition that online degree programs have now made higher education more easily accessible to a wide 

variety of people. Indeed, there is still concern for issues related to the digital divide (Lax, 2001; Mack, 2001) 

and the fact that worldwide, those with more access to wealth and power are also likely to have easier access to 

technology, therefore increasing the gap between the haves and the have-nots. However, more recent 

discussions have focused not only on analysis of the digital divide, but also on ways of theorizing and 

intervening in practice to attempt to dismantle it (Gorski, 2002). 

Given the proliferation of online degree programs, a number of researchers in recent years have undertaken 

various studies about the learning process online and its related 

areas. For example, Daley and colleagues (2001), studied five online adult education graduate classes from five 

different universities, and how 45 individual students reported learning. They evaluated students’ remarks 

according to a five dimension learning model discussed by Marzano and Pickering (1997) that includes 

consideration of: attitudes and perceptions toward learning; how they acquire and integrate knowledge; how they 

build on or extend it; how they use it meaningfully in their lives; and “types of thinking or habits of mind” 

(Daley et al., 2001, p. 135). Students’ reflective thinking and integration of learning were dependent on how 

positively they felt about technology, which obviously affected their participation level, engagement and use of 

the technology, and interaction among learners. 

While the focus of this study was more on the learning process of these 45 individuals, others have emphasized 

the importance of building community online as central to the learning process. Highly relational learners have 

sometimes found the “faceless” dimension of online learning somewhat impersonal and problematic in meeting 

their learning needs, and have tended to be the ones to drop out of programs unless there was a way to have their 

relational learning needs met. As Tu and McIsaac (2002) suggest, those who experience a high level of what 

they refer to as “social presence”—the extent to which one feels that there is awareness of real communication in 

its multiple dimensions with others in an interaction in a particular context—report greater satisfaction with that 

context. This is true in online environments as well as face-to-face environments. Of course, in online 

environments the non-verbal dimensions of communication, such as facial expression and eye contact, are 

typically missing. Thus,Tu and McIsaac (2002) studied the specific ways 51 students experienced a high level of 

social presence in the online learning environment, and evaluated their results across three dimensions: social 

context, online communication, and interactivity. Participants valued the online social context that promoted 

familiarity with other participants, trust, informal communication as well as communication about the subject 

being studied, and a positive experience and attitude toward technology. Online communication was positively 

valued if it was expressive, included affect and emotion, and was easy to understand. Interaction that was 

immediate, invited a response, and discussed familiar topics yielded a high level of social presence. These 

factors were able to provide a satisfactory substitution for the fact that FTF communication was missing. And of 

course, participants with strong computer skills, and the availability of immediate technical support were factors 

that added to satisfaction with online learning. 

While Tu and McIsaac (2002) specifically studied “social presence” online, others have considered how students 

interact online, and how to build community in CMC environments to facilitate student learning. Creating 

collaborative learning opportunities among smaller groups of students is a significant way of building 

community online (Bernard, Rojo-de-Rubalcava, & St-Pierre, 2000). Further, several researchers report on case 

studies at particular universities that support the notion that collaborative learning opportunities increase a sense 

116

of community among online learners (Barab, Thomas, & Merrill, 2001; Fischer & Coleman, 2001-2002; Murphy 

& Cifuentes, 2001). 

There has been much consideration of online communication, how to create community, especially through the 

use of group assignments and other collaborative learning activities; nevertheless, the literature on cohort 

learning in online degree programs in cyberspace is extremely limited. While it appears that some of the 

classroom cases discussed in the literature and cited above may perhaps have taken place in cohort programs, in 

most of these cases, there is no consideration of the cohort experience itself. Mason (2000) did discuss a cohort 

of student teachers’ computer mediated communication, but her focus was on how CMC was used to enhance 

students’ face-to-face learning rather than either online learning or the cohort experience. Strohschen and 

Heaney (2000) have discussed the role of team teaching and learning and some cohort dynamics in attempting to 

implement a critical pedagogy approach in an online degree program, which offers an important beginning to 

understanding how online cohorts can facilitate learning. While their focus was more on the team teaching aspect 

than the cohort experience per se, they did point out that student cohort members formed communities for both 

the goals of building a collaborative learning environment, as well as for meeting their relational and affective 

needs. 

Given that cohort programs have met great success in face-to-face degree programs made up of adult learners, it 

is indeed surprising that there has not been more consideration of cohort learning specifically in online 

environments. Lawrence (2002), Bochenek (1999), Saltiel and Russo (2001), and Brooks (1998) have all 

discussed the benefit of the cohort learning experience in adult degree programs. Nesbit (2001) has noted that 

cohort groups often facilitate transformative learning experiences among adult learners in cohort degree 

programs. In this sense “transformative learning” appears to be based on Jack Mezirow’s (1995) theory of 

transformative learning, which focuses on how adults learn and “transform” through the process of critically 

reflecting on their underlying assumptions, thereby developing a broader and more inclusive view of the world, 

and of how people of multiple backgrounds, cultures, and social groups, learn, grow, and change, Following a 

similar line of thinking to Nesbit (2001), Scribner and Donaldson (2001) discussed a study of a cohort group of 

educational leaders in a doctoral program, and the role of the cohort in facilitating transformative learning. In 

addition, Fleming (1998) has studied the power of the community experience and transformative learning in 

residential adult learning settings, some of which were cohort based. 

It is clear that insights gained from cohort learning can easily be translated to online degree programs. As noted 

above, there are clearly online degree programs that are conducted in cohorts, though at this point, the literature 

specifically on the cohort online learning experience itself, particularly that includes the voices of student 

participants, is absent. Thus this study gives voice to both the student and faculty participants. 

Method 

This study was a participatory action research study where, we, as cohort members in the master’s program in 

adult education, conducted research about ourselves and our own experience of cohort learning online within this 

program at a university in the U.S. Midwest. There were 10 participants in the study, nine women and one man; 

one of us is African American, the rest are European-American. Seven of us were regular members of a cohort of 

eight. Our eighth member had taken most classes with an earlier cohort, but joined us to complete the last 

semester, the Integrative Seminar, to complete the program. The other two of us were co-instructors teaching 

both the first and last semesters of this cohort online master’s program. 

From a design perspective, this participatory action research project was conducted within the case of a single 

cohort group, where all members of the group had participated in the cohort experience. From that standpoint, it 

was a case study approach of those of us who shared in the cohort experience through the final term of the 

program, when those of us who were students were graduating from the program. As both Yin (1994) and 

Merriam (1998) have suggested, a case study is a naturally bounded system, such as a classroom or an 

organization, where participants in the case are obviously members of the naturally bounded system. The 

purpose of a case study is to explore the particular in depth. It is not to make generalizations; rather, it is up to 

the reader to determine whether or not the insights gained from studying the particular case in depth can be 

applied to other settings. Thus, our purpose here is to offer the exploration of our own experience together. It is 

up to other educators and learners who are trying to implement online pedagogy that meets the needs of both the 

educators and the learners to determine if it is applicable. 

117

Not only is this a case study, it is a participatory action research case study. As Merriam and Simpson (2000) 

have noted, participatory and action research is research conducted by participants specifically to make 

something happen. In this case, as participants in the program, we used the concluding seminar to study, write 

about, and facilitate the integration of our own learning, and particularly for the eight of us who were students, to 

give voice to our own experience of learning online, specifically in a cohort experience. In the process of 

studying our own learning experiences, we made new meaning of both the cohort experience itself and of our 

own ongoing learning experiences throughout the program. Ultimately, we also did a presentation at a USA 

research conference, and have crafted this article together. The process of doing it has helped make further 

meaning of our experience together. 

Theoretical Framework 

Both the online master’s program and the participatory action research study itself were informed by a social 

constructivist theoretical framework. Fosnot (1996) explains that social constructivism as an educational theory 

“construes learning as an interpretive, recursive, building process by active learners interacting with the physical 

and social world” (p. 30). In essence, such a view of learning is based on the assumption that people learn best 

when they apply theory to practice, then revise the theory in light of what is learned in the application, and 

reapply the revised theory. It also means exploring the unstated theory that is imbedded in practice, which 

typically happens both through dialogue and critical reflection (Brookfield & Preskill, 1999), which suggests that 

learning is a social process. This theory to practice looping through reflection, dialogue, and application that is 

an important part of reflective practice is the way the online master’s program is structured, and is how the 

program is informed by a social constructivist educational framework. While we believe that it is possible for 

learning to occur through other teaching methodologies, such as the lecture method or simply reading 

information on one’s own, a basic assumption of a social constructivist framework as applied to this program is 

that learning is a social and interactive process. 

The participatory action research itself was also grounded in a social constructivist view of research. As 

Schwandt (2000) has stated in applying constructivism to research, “We invent concepts, models and schemes to 

make sense of experience, and we continually test and modify these constructions in light of new experiences.” 

We do so “against a backdrop of shared understandings, practices, language, and so forth”(p. 197). This is 

precisely what our view of the participatory action research process. In our process together, we developed 

concepts in light of our experiences, and then we refined them as we made ongoing meaning of this process. 

Thus, this constructivist paradigm of research was relevant to this participatory action research study, because it 

was compatible with the basic beliefs that learning happens in a socio-cultural context, and new knowledge is 

constructed in light of dialogue, in light of challenging one’s assumptions through reflection, and in light of 

one’s past experience and new experience of putting ideas into practice. This will be explored more fully in our 

discussion of the data collection and analysis process, which will make more sense following our explanation of 

how the master’s program is structured. 

The Adult Education Online Master’s Program 

The Adult and Continuing Education (ACE) Department launched its online master’s graduate program (AOP) 

in adult and continuing education in 1998. This Internet-based graduate degree program was modeled after its 

highly successful cohort-based face-to-face program, which is also theoretically grounded in a social 

constructivist view of education. In an effort to bring key features of the face-to-face program “online,” the 

design created by faculty of the ACE Department sought to replicate the participatory, interactive instructional 

strategies and the cohort-building activities in the context of technology. 

Because a social constructivist framework is grounded in the belief that learning is a social process with 

reflective dialogue as one of its primary components, the Internet-based program design, therefore, emphasizes 

community building with several features. Students attend an initial two-day residential learning experience, 

where they come together at a conference center. This face-to-face residential serves as an introduction to the 

program’s faculty and students; they are also introduced to the educational philosophy of cohort-based learning. 

While participation in the residential is a requirement and its costs are part of student fees, students out of the 

country have the option to “attend” the residential via submitting a video in which they introduce themselves and 

share their expectations; the group at the residential, in turn, makes a short video of the participants and sends it 

to the out-of-the-country participant. In situations where a cohort group has “met” others in this way, faculty 

advisors have intensified email communication to everyone and also encouraged the creation of student websites. 

118

Since a 4-hour training seminar on the use of the program’s web-based software is a part of the residential, those 

students who could not attend, received written instructions and one-on-one support from faculty and help desk 

staff at the university. 

To foster interactive participation, two courses each semester are integrated and taught by a faculty team of two. 

The faculty team creates a combined syllabus that allows course projects to dovetail content areas of the courses. 

Course sequences are selected to fit thematically and deepen research opportunities each term. For example, 

“History and Philosophy of Adult Education” is integrated with “Adult Education in a Sociocultural and Political 

Context” so that the exploration of the development of the field and its current trends and issues are studied 

during the term. Moreover, as a faculty cohort, instructors model dialogue and collaboration and demonstrate 

various perspectives on discussed topics. Students are expected to discuss readings from a theoretical 

perspective, to talk about how the ideas relate to their life experiences, and their practices. They are also 

expected to develop their own philosophy and practice of adult education throughout the program, based on 

ongoing reflection and application of ideas. Success each term is determined by how well students demonstrate 

their ability to do this in their online discussions and in their course papers. In addition, students are expected to 

complete an in-depth independent inquiry project (under the guidance of a faculty member) during a four month 

independent study term, This extended online inquiry phase allows students to engage in more in-depth research 

of their preferred topics within the field of adult education. Topics of inquiries vary greatly. Students have 

developed a series of stress management courses for law enforcement staff, Freirean-based prenatal education 

classes, and an online program design strategic plan for a community college, to give but a few examples. 

Faculty teams are expected to follow the same format for discussion boards and use of the technology so that 

students don’t need to learn a different format or set of navigation skills each term. Course content is introduced 

by a posted syllabus, texts, and suggested study activities and course assignments. Jointly, the faculty and 

students negotiate the final syllabus and outcomes. An example is this study, which emerged as the final product 

for the last term of the program, the “Integrative Seminar.” Furthermore, since incorporating student 

background and content knowledge is crucial to an experience-based knowledge construction model, the format 

of asynchronous online board discussions makes room for adaptations in the flow of discussion and topics 

covered within stated course objectives. 

The program director guides each cohort through their asynchronous learning journey, which, aside from the 

opening residential, takes place fully on discussion boards. The AOP has used a number of web-based software 

over the years (e.g. TopClass, Discuss, and WebCT) with the same results: a 97% completion rate has been 

achieved in the program to date. The essential technology needs center on use of a discussion board, emails, and 

synchronous chat rooms for informal discussion and sharing of ideas and personal information. The chat room 

and a “lounge” area (i.e., a portion of the discussion board dedicated to talks outside of class work) serve in 

much the same way as the cafeteria or hallway in an on campus class… they allow students to nurture their 

affective support systems if they so choose. No other technological features such as white boards for 

synchronous instruction are used. However, periodically, the need by students has surfaced to receive hard 

copies of articles or faculty feedback and occasional telephone calls. 

Data Collection and Analysis 

While our own participatory action research design was heavily informed by the theoretical framework of social 

constructivism, from a practical perspective the data collection and analysis process was based on Carr and 

Kemmis’s (1986) action research cycle of planning, acting, observing, reflecting. Our planning process included 

generated ideas about how we would go about analyzing our online experience as a cohort of learners and 

teachers. In this process we decided we would communicate online about our particularly salient points of 

learning in the cohort experience. In preparing to do this we agreed that we would re-read some of our online 

comments over the course of the 15 months, and each write a reflection on primary moments, events, and 

discussions that stood out. Next, we acted by actually writing online. Thus, the primary means of data 

collection was our ongoing ONLINE discussion of readings about online education, review of key readings 

about adult development and learning in the program, and their application to our lives and educational practices. 

Next, the entire group, both participants and instructors, observed the common themes in our writing, and 

together we began developing some categories. In essence, this generation of common themes and categories is 

reflective of the constant comparative method of data analysis as discussed by Merriam (1998), but within this 

participatory action research process, this generation of data analysis and telling our data story was ongoing. 

119

In the next phase, each of us agreed to take responsibility for interviewing others over e-mail in more depth 

about a particular theme or category related to our learning, and then writing up those sections for the rest of the 

group to read and comment on. This was part of the reflection phase. We then commented on each other’s 

sections and clarified and made further refinements on the points we had made as individuals in order to prepare 

for the final write-up of our experience. This process of seeking clarification might be akin to member checks, 

which are strategies typically used in qualitative research to enhance the dependability of findings (Lincoln& 

Guba, 1985; Merriam, 1998). We then wrote up our final piece. 

Part of the action part of this participatory action-research project was that we also opted to present our work 

together at a research conference. The findings that are presented in this manuscript are an expansion of what 

was presented at the conference, and our own further reflections on this knowledge construction process. In a 

sense, participatory action research is always ongoing. This article presents the results of our collective work as 

we reflected on how our online experience unfolded, and the knowledge that we created as a result of our 

experience together. 


As noted above, a social constructivist theoretical framework assumes that knowledge construction is ongoing 

and takes place in a social context as ideas are discussed and learning processes are negotiated, and is related 

both to the content and the process of learning. Thus we’ve opted to break down our findings into two primary 

areas that reflect this paradigm 1) the ongoing negotiation of the learning process, and 2) the ongoing 

construction of knowledge by both individuals and the group as a whole. Permeating each area of the findings 

is the realization we gained that the Internet-based delivery format, i.e., the “technology”, was but the vessel 

within which we engaged in our studies and action research. While the technology context, this “container”, 

inextricably helped and hindered our learning and knowledge construction processes, it simply was the medium 

for learning and not the primary learning itself. 

The Ongoing Negotiation of the Learning Process 

Central to the first area of findings related to the ongoing negotiation of the learning process were three primary 

categories: a) the importance of the residential; b) a consistent but flexible cohort structure , and c) building 

ongoing relationships. 

The Importance of the Residential 

The two-day residential at the beginning of the program was deemed as central to the success of the program. 

Several participants consider the residential as perhaps the most significant factor in contributing to the success 

of the online experience. The residential consists of structured formal and informal activities. Primary to 

students’ interest as they enter the program is the need to become familiar with the technological aspects of 

online communication, to get clarification of expectations for participation, and to understand the ways 

validation of completed assignments via distance is accomplished. A key element not generally expected by 

students surfaces during discussions at the residential: the focus on community building and co-learning, which 

seems to be exacerbated in online discussion with its missing “face-to-face element of non-verbal 

communications. Students acknowledge that meeting the cohort and faculty members allowed them to respond to 

more than “words on a screen,” in the words of a participant. Through life history presentations and informal 

dialogue after classes, which is later maintained in the lounge and with chats, an atmosphere is created that 

makes space for trust building. As Mike put it, “It was when a professor sat down with me on the steps of the 

building that I knew I would feel comfortable in this program.” The residential at the end of the program 

elucidates the importance of this affective aspect of learning. Students report that their own growth and 

habilitation into knowledge-producing scholars became clear to them as they progressed through the program. 

While faculty teams change each semester, the students increasingly rely on one another’s voices on the screen 

to verify, validate, and critically reflect individually and in the group on their respective contributions to the 

discourse on the given topics. The residential sets the tone for negotiated co-learning and establishes an 

interdependency of the roles of “teacher and student.” 

120

A Consistent but Flexible Cohort Structure 

The components of the program structure included clearly announced course requirements with the 

encouragement of tailoring each term’s topics to include students’ experiences , a more or less “closed” cohort 

group with only occasional additions of “course takers”, and the flexibility of negotiating course assignments 

and alternative communication when technological problems closed the board. Prior to the start of each term, a 

syllabus for the integrated courses was posted. Members of the cohort had the opportunity to review the syllabus, 

its proposed texts, and suggested assignments and negotiate for changes or adjustments. Suggestions have been 

minimal, but changes were negotiated easily. Robin noted, “It was more that it was out there if we chose to 

negotiate…the feeling that you can help control your own learning experience is very empowering.” 

The cohort remained intact throughout the program, in that those members that began the program together were 

still together the last term. The relationships, which began at the residential, developed throughout each term. 

Pam noted, “As the courses progressed, members learned a great deal about each other. An atmosphere of 

mutual respect prevailed as discussions sometimes became intense when members shared opinions, feelings and 

personal experiences. Life changes occurred in the cohort including a birth, medical problems, family issues and 

professional crises including job changes.” During the last term one new member, who had started the program 

with an earlier cohort, was added to the group, and became a part of the group relatively easily. Marguerite 

notes, “…it helped that he was in an earlier cohort”. Thus, the cohort is mostly a closed group, but the structure 

is flexible enough to include an occasional student from another cohort. 

There are always some technological problems in online delivery of programs. During the residential, “the 

technology”, i.e., computer skills and managing primarily text-based learning, was cited as one of the main 

concerns upon entering the program. A portion of the residential was spent on training to use the particular 

software platform with the urgent reminder that any problems can be overcome with adequate communication. 

Nevertheless, many members expressed frustrations they experienced in dealing with various aspects of Internetbased 

courses. This was often based on their limited familiarity with Internet navigation or of computer skills. 

Mary describes her experience using the Mac computer, “Once I learned how to navigate – by trial and error in 

the system, the technology was easy.” Marguerite “needed additional technical support and paid a private 

consultant.” As the program progressed, however, many of these issues were resolved. From the faculty’s 

viewpoint, such issues were minimized with open discussion on our differences in computer savvy. Mary Lynn’s 

comment puts the issue in perspective, “They (technology problems) are to be expected in this kind of 

environment.” The availability of help desk staff as part of the university infrastructure and assurances by faculty 

when a student struggled with technology put it into perspective – that of being a vehicle to communication. 

Most of the students’ initial concern with the technology had abated by mid semester of the first term, though 

there were occasions when we had to resort to e-mail when the server was down. Dealing with these occasional 

difficulties was kept in perspective. 

The Ongoing Building of Relationships 

The ongoing building of relationships among and between students and faculty, in the context of sharing the 

roles of teachers and learners, was another important factor in the ongoing negotiation of the learning process. 

Relationship building became an integral part of the online learning experience. Some noted that until the last 

term, the fact that the cohort group consisted of all women might have added to the significance that 

relationships played. Mary noted “We found common ground in the experience of being female. It became easy 

and comfortable to reveal deep feelings about the adventures (positive and negative) of learning as a woman… 

While we bonded well as a group of women, there were no adverse results when a man joined the group.” In 

reflecting on joining the group the last term, Mike noted, “having the support of the cohort in doing so helps 

greatly. It’s like walking into a room of people you don’t know…if there is acceptance, you can get on with 

business more easily than if there is resistance.” And Janet emphasized, "Every good relationship is based 

around some commonality" and there was the commonality of a cohort learning experience online. 

There were a number of collaborative assignments, and participants noted that these were an opportunity to 

develop friendships via telephone calls, the chat room, and emails. Janet observed, “We learned to value the 

differences in perceptions based on age, race, politics, culture, marital status, life experiences, and the last term, 

gender, because they became tools for learning. Our various backgrounds and ways of knowing made the 

learning very broad." 

121

An obvious factor in online relationships is that no one can attend class and not participate. Pam pointed out , 

“Everyone had to find her voice. I feel that I know people far better than I might have in the less intense situation 

of a classroom. " And Mary explained, “without inflection, tone, body language or eye contact, our words and 

stories built our relationships.” Sharing one’s personal story and life experience relative to the discussion of 

readings was not only an important part of relationship-building but also the theory-practice connection of how 

one applies what one is learning in practice. Mary summed up the group’s experience, “We concluded that the 

power of online learning includes building relationships. Criteria for success are 1) meeting at the residential, 2) 

remaining together throughout the program, 3) being open, honest and participatory in postings and feedback, 4) 

respecting and learning from the diversity of experiences and opinions. These factors produced a comfortable, 

supportive, trusting, and productive group relationship.” 

The Ongoing Construction of Knowledge of Individuals and the Group 

The cohort experience was particularly important in regard to the second area of findings, the ongoing 

construction of knowledge by individuals and the group. These findings fell into three primary areas discussed 

below: a) Team teaching/cohort learning as contributing to transformative learning; b) the connection of theory 

to real life practice; c) the value of group support in conducting research and constructing knowledge together. 

Team-Teaching—Cohort-Learning as Contributing to Transformative Learning 

Nearly all members considered their online learning experience to be transformative , in how Jack Mezirow 

(1995) defines the terms, where they constructed new knowledge together. Marguerite, in thinking about the 

cohort noted, “I have developed personally and professionally through my exposure to their ideas, their 

contributions, their passion and compassion." Cohort members provide a continuity, and yet a diversity of 

voices, and Marguerite went on to explain “our cohort provided balance in the voices of teachers and students. 

Each cohort group is a unique blend of personalities and fields of experience, which can challenge concerns 

about isolation with online learning.” Mary Lynn emphasized the professional development of the cohort and 

noted “Professionally, I was exposed to more areas of the field of Adult Education than just my own practice 

areas," and Mary explained "Learning in a cohort was transformative because it became so much more 

comfortable to professionally facilitate different views in a classroom." Speaking specifically on the issue of 

constructing knowledge together Mary Lynn noted, “Having the opportunity to reflect and build on the 

contributions of others made the knowledge I constructed more powerful. I liked the asynchronous discussion 

board…" 

In general, the team teaching dimension of this online program was experienced positively and as another 

potential avenue to interact with new ideas. But as Pam noted, "Team teaching was very effective when the 

team members both took part. Some faculty handled the online classroom better than others." Mary Lynn 

explained that the team teaching "In most cases, was great because we got the benefit of different 'expert' 

perspectives on the same topic," enabling them to construct knowledge in new and deeper ways. Mary referred 

to the benefit of the residential in meeting instructors personally and face-to-face. "I definitely felt more bonded 

with and had a sense of relationship with the instructors I'd met personally. The face-to-face heightened my 

awareness of the gift of our senses and the value of seeing, hearing, and touching. I felt I knew them, and they 

me." 

From the vantage point of this faculty team , the fact that we had developed a collegial relationship prior to 

teaching online together added to our ability to work well as a teaching team both the first and last semesters in 

this online program. Even though we are quite different in our approaches, with one of us being much more 

grounded in philosophy and more open ended on assignments, and the other more grounded in sociology and 

more detail oriented, we both participated equally online and appreciated the other’s different perspectives, and 

pushed the students to do their best work, and to see themselves as adult education scholars that can contribute 

to the knowledge base of this field. 

Connecting Theory to Real-Life Practice 

In explaining the connection of theory to real life practice and its role in knowledge construction, Janet explained 

“First, we had to examine our primary purpose for coming together, which was to study the theories and 

practices…to develop our own philosophies about Adult Education and how they relate to real-life practice. 

Secondly, we had to find ways to transfer the value of the experience of learning in an online cohort and how 

that experience relates to practice.” Janet goes on to explain the importance of learning enough adult education 

content “in order to begin to develop our own theories.” The knowledge has changed Mary Lynn’s practice as an 

122

adult instructor in that she is “more careful when building thematic units and curriculum changes to include real 

life contextual materials. “ She goes on to explain that, “ I am more conscious of silence during discussions and 

let the students help each other draw out their own experiences more. I have stopped using 12-year old reading 

texts and moved forward to more relevant materials.” When asked about considerations in negotiating planning 

with others who may have differing philosophies on adult education, Robin felt that colleagues needed to be 

reminded that these are adults, who may be subject matter experts in their own field, and who are probably selfdirected 

learners in that field specifically. Because of these three factors, she felt that, “This means a change in 

the way an educator, views her/his role in the process of education, and also a change in the way material is 

presented, negotiated and the way grades are assessed.” 

When examining the impact of learning in an online cohort and how that experience relates to practice, Janet 

explains, “We were better informed about the knowledge because of time that online learning allowed for intense 

and well thought out interaction with the content. Yet, what was most powerful for all of us was that the cohort 

provided the safe environment for permitting us to take a hard and honest look at our own knowledge. Therefore, 

the online cohort experience added great value to each individual’s ability to determine either ‘how they are’ in 

their practice or ‘how they plan to change’ their practice.” 

Group Support in Deeper Knowledge Construction 

Robin explored the role of group support in enabling the group to go deeper into the material and also conduct 

and complete research projects, and explains “The cohort became a place for some of us to bounce ideas off of 

one another and as a place to build a team to help in research.” Some focused more on support for creating and 

accessing knowledge, while others gained more from the emotional support and encouragement to go deeper. 

Robin notes that the cohort was valuable in the research and knowledge construction process in “knowing they 

were writing to an audience, an audience that they knew in this case, helped them hone their work. Still others 

took what those around them researched and used it to springboard their own ideas.” 

While virtually all members used the cohort for both intellectual and emotional support, the research process 

itself is often a solitary process, and many of their comments reflected this, particularly in regard to the research 

term. As Janet explains, “As much as I enjoyed the cohort, I felt that I did my research projects mostly alone. I 

felt that when we had collaborative book reports, each on a different aspect of the study, that reading from the 

others was a wonderful way to not have to do so much research and to receive the benefit of other's research. I 

guess they aided just by my knowing who would be reading my project and feeling that I wanted my work to be 

respected by them.” Mary Lynn also noted, “I learned a lot from the others in my cohort, but none of it related 

directly to the research for my inquiry. I completed that research on my own.“ Pam also expressed this although 

she didn’t find it problematic and explains, “I had no problem with doing the research on my own. If I had 

needed help or aid from the cohort, I would have felt comfortable asking for it.” And Robin noted, “I didn’t 

really feel a connection with my cohort during the inquiry project. They all chose such different topics from 

mine that I really didn’t feel I could ask them for help.” While some of this was more or less expected, given 

that aspects of research are a somewhat solitary process, it was clear that they had a different experience of 

conducting this participatory action research project than they did of their inquiry term when they were 

conducting an independent research project. But both of these aspects were ways of constructing both individual 

and group knowledge. 

Discussion 

Not unlike other technology vehicles of communication that the last century brought to society, online 

technology is but a tool. The study suggests that while it is very important to build a consistent and flexible 

structure that can accommodate technology in the building of community in online environments, it is helpful to 

draw on what is already known to facilitate successful adult degree programs. As many have pointed out, cohort 

programs have been extremely successful adult learning environments, precisely because the continuity of the 

cohort group from one course to another builds a stable community (Lawrence, 2002; Saltiel & Russo, 2001). It 

is because of this stable community that the limitations of technology, such as when the server would go down, 

were relatively easily dealt with by this cohort group. In these kinds of circumstances, the group would take 

some responsibility for deciding how to deal with the problem, such as resorting to e-mail, phone calls, until the 

difficulty with the usual mode of communication was taken care of. Of course, the availability of instructors 

and/or the director of the program was important. But as a community of adult learners that has grown to know 

and count on each other over time, as a group, we proved to be quite capable of determining how to solve these 

123

difficulties as they arose. The social constructivist theoretical framework of the program design probably 

enhanced the ease with which the group took action. Given that learners were expected to take some 

responsibility for their learning through critical reflection and in applying ideas in practice relative to the course 

content, it was easy to critically reflect and take action to solve practical problems, such as the few times when 

the usual mode of technological communication didn’t work. 

Not only was the cohort feature central to the success of the program, as the recent-graduates/co-authors of this 

paper note, the design feature of the residential where most of the participants came together to meet each other 

face-to-face at the start of the program, was integral to it. As Fleming (1998) also found in her study of 

residential learning, the residential aspect helped participants get to know each other in a face-to-face setting, 

which helped build a the base for a solid sense of community at the beginning of the program. While we 

recognize that it may not always be practical or possible to have an entire group come together at the start of a 

program, we do believe that it is possible to provide some face-to-face contact at the beginning. This might be 

through requiring, or strongly encouraging those who can, to physically attend, and/or for those others who are 

unable to attend to make themselves present in some face-to-face way. This might be through interactive video, 

or sending videos back and forth, or through a combination of picture and/or audio contact. 

The feature of the beginning residential likely increased the group’s ability to maintain a high level of what Tu 

and McIsaac (2002) refer to as “social presence” throughout the 15 months of the program. The high level of 

“social presence” was also enhanced by the fact that the structural components of the program and a commitment 

by the teachers to participatory pedagogy in light of the social constructivist theoretical grounding of the 

program fostered the creation of a learning community that allowed cohort members to engage in sharing power. 

Between teachers and students, the concept of negotiating components like assignments, readings, group 

projects, and quality and quantity of weekly postings were initially introduced at the residential. While the 

community was established based on the common purpose of working together in a cohort, the relationships 

were built because of everyone’s commitment to trust and participation. In addition, in teaching teams each 

term, teachers modeled a commitment to collaboration, much as they provided a structure for collaboration to 

take place. While this was more successful with some teaching teams than others, the openness of teachers to 

adjust discussion to its flow and emerging themes or to extend assignments deadlines when technology failed or 

skill levels varied added to the comfort with which everyone was able to reach agreed upon objectives. The 16week 

independent inquiry phase allowed for mentoring and the picking up of themes in one-on-one exchanges. 

Throughout the program, as a collective group, we nurtured community building through acknowledging 

technological and other limitations and through expressing experiences within our affective and cognitive 

domains. This fomented an environment in which the cohort group could engage in collective knowledge 

construction. 

Knowledge construction online appears to have occurred within an integrated set of variables, i.e., individual and 

group processes within affective and cognitive domains, and with an instrumental dimension related to being 

able to navigate Internet-based course delivery and troubleshoot when problems with technology constrained the 

flow of communication. While this is no great revelation as the existing literature tells us (Barab, Thomas, & 

Merrill, 2001; Daley et al., 2001; Northrup, 2002), an additional element comes into play that can be described 

as an awareness of being connected – in both its technological and metaphorical meanings. Indeed, technology 

skills are necessary to navigate; yet it is important to have a way to negotiate the limits and failures of 

technology, such as when access to the discussion boards fail. But as this participatory research project brings to 

light, the social and relational elements of community building and collaboration are instrumental in energizing 

the interaction of the variables discussed above. 

Cohort members expected to gain “an education” within the convenience of an Internet-based, distance 

education program. The aspect of their personal transformation as a part of this learning was not expected nor 

was it clear at first. In fact, it was not until the end of the program and during the reflection on the experiences 

that cohort members expressed their realization that they “had grown” more than professionally. The knowledge 

that had been collectively created was not necessarily “new” as much as it had become emotionally and 

cognitively integrated into awareness of what it means now to “be an educator” and to “be a whole person.” The 

awareness that the role of educator is inextricably connected to the person paved the way to creating entirely new 

approaches to each one’s practice. 

In this regard, the connection of theory to real life practice took on new dimensions. Cohort members expressed 

that they gained a sense of strength and validation of hunches they have had all along. Moreover, by being able 

to learn from one another about best practices, successes, and failures, cohort members gained from the 

authenticity of the “real life” experiences their colleagues shared online instead of relying primarily on “text 

124

ook” theories as guides. These exchanges, which had been structured into assignments and course discussion, 

are invaluable in bringing to life the tenets of participatory adult education practice – in any delivery setting. 

Additionally, the program assignments required that cohort members apply what was discussed and/or learned 

from texts and teachers with projects or outside observations of educational settings. In this way, analyses and 

syntheses of theories were immediately applied to their real life work settings. Indeed, as many have discussed, 

sharing real-life experiences, and the discussion of application of theory to real-life practice seems to be common 

to most higher education cohort groups (Lawrence, 2002; Saltiel and Russo, 2001); it is surely not unique to 

cohort learning online. However, as the experience of this cohort indicates, these features that are often typical 

of face-to-face cohorts can be translated to the online environment. Perhaps the primary difference is only in the 

immediacy with which one could apply an idea and get feedback. Once a “new idea” was applied, the online 

exchanges provided opportunities for sharing insights and reflections on the experiences quite immediately. 

Given the structure of the program, the flexibility of instructors to adapt “lessons” each week to the occurrences 

of insights, and make the space that the informal areas of the discussion board offered (e.g. a lounge and a chat 

room), group support became an integral aspect of the knowledge construction. At any time, one could turn to 

the board and with that to one another to bounce off ideas or share frustrations and get a relatively immediate 

posting of support or feedback. In this way, no cohort member was ever “alone” even when independent research 

tasks or course work had one working in Georgia and another laboring over an assignment in Italy. While Daley 

et al. (2001) also touch on this idea, they discuss it from the perspective of those who have positive attitudes 

about the use of technology, which perhaps implies interactions and constructing knowledge together. But in 

this study, this sense of bringing theory and practice together was a result, in part, of our collaborative 

interactions as a cohort. 

What does this suggest for the practice of education in Internet-based degree program delivery? Obviously as a 

group, we first would strongly recommend the following: 

1. use of an ongoing cohort learning model; 

2. an opening residential, or some other means where participants can meet each other face-to-face at the very 

beginning of the program, to develop a sense of a commitment to a learning community; 

3. a commitment by instructors to the use of a participatory pedagogy; 

4. a specific strategy for learning to both initially use the technology, and a way to trouble shoot it; 

5. collaborative assignments with specific but negotiable guidelines. 

It seems that it is these five components together that contributed to the success of the program and to the 

members’ learning. While it may not be possible to include all of these components in other programs, or to 

always have the benefit of a teaching team and two integrated courses, it is possible to incorporate many of these 

features, that are likely to contribute to the success and overall learning of a cohort group. Perhaps most 

important to this particular group was the willingness of all participants (both faculty and students) to recognize 

that we are all simultaneously teachers and learners. Of course, faculty are responsible for designing a syllabus 

and helping organize the discussion. Yet these faculty also respected the fact that most of the students were 

adult educators themselves, and in teaching roles in their day-to-day professional lives on a regular basis. In 

these courses, they were in teaching roles as well, as fully active participants in discussions, and in the online 

presentations that they were responsible for. Online, everyone’s contributions “stand in front of the class,” and 

syntheses and analyses of concepts and ideas become a collective and collaborative means of knowledge 

production by the cohort group. This is what makes it “constructing knowledge in community “ and is the heart 

of the cohort experience. In sum, the implication for developing a cyber pedagogy is quite simple: it is found in 

the authentic voices of the learners as they collaboratively create knowledge and self-determine personal growth 

in a community of trust and mutual support in a cohort setting. In that, a cohort-based approach to collaborative 

learning and knowledge construction grounded in participatory education principles works in spite of the 

constraints of “technology.” 

References 

Barab, S., Thomas, M., & Merrill, H. (2001). Online learning: From information dissemination to fostering 

collaboration. Journal of Interactive Learning Research, 12 (1), 105-43. 

Bernard, R., Rojo-de-Rubaclava, B., & St. Pierre, D. (2000). Collaborative online distance learning: issues for 

future practice and research. Distance Education, 21 (2), 260-277. 

125

Bochenek, M. (1999). Classroom communities: Successful learning design or another path to chaos? Academic 

Exchange Quarterly, 3 (3), retrieved September 19, 2001 from 

http://www.newcastle.edu.au/department/ar/AEQ/V3-3/V3-3-Focus-2.htm 

Brooks, P. . (1998). Cohort communities in higher education: The best example of adult education. Retrieved 

September 19, 2001 from Adult Education Research Conference web site: 

http://www.edst.educ.ubc.ca/aerc/1998/98brooks.htm 

Brookfield, S. D. & Preskill, S. (1999). Discussion as a way of teaching. San Francisco: Jossey-Bass 

Carr,W., and Kemmis, S. (1986). Becoming critical. London: Falmer Press. 

Daley, B., Watkins, K., Williams, S. Wall; Courtenay, B.; Davis, M. & Dymock, D. (2001). Exploring learning 

in a technology-enhanced environment. Educational Technology & Society, 4 (3), 126-38. 

Dede, C. (1996). Distance learning—Distributed learning: Making the transformation. Learning and Leading 

with Technology, 23 (7), 25-30. 

Fleming, J. (1998). Understanding residential learning: The power of detachment and continuity. Adult 

Education Quarterly, 48 (4 ), 260-71. 

Fischer, M. & Coleman, B. (2001-2002). Collaborative online learning in virtual discussions. Journal of 

Educational Technology Systems, 30 (1), 3-17 

Fosnot, C. T. (1996). Constructivism: Theory, perspectives, and practice. New York, Teachers College Press 

Gorski, P. (2002). Dismantling the digital divide: A multicultural education framework. Multicultural Education, 

10 (1), 28-30. 

Green, K. (1999). When wishes come true: Colleges and the convergence of access, lifelong learning, and 

technology. Change, 31, 10-15. 

Lawrence, R. (2002). A small circle of friends: Cohort groups as learning communities. In S. Imel (ed), Adult 

leaning in community (83-92). New Directions for Adult and Continuing Education, no. 95. San Francisco: 

Jossey-Bass.. 

Lax, S. (Ed) (2001). Access denied in the information age. New York: Pelgrave. 

Lincoln. Y., & Guba, E. (1985). Naturalistic inquiry. Thousand Oaks, CA: Sage. 

ack, R. (2001). The digital divide: Standing at the intersection of race and technology. Durham, NC: Carolina 

Academic Press. 

Marzano, R. J., & Pickering, D. J. (1997). Dimensions of learning (2nd ed). Alexandria, VA: Association for 

Supervision and Curriculum Development. 

Mason, C. (2000). Online teacher education: An analysis of student teachers' use of computer-mediated 

communication. International Journal of Social Education, 15 (1), 19-38. 

Merriam, S.B. (1998). Qualitative research and case study applications in education. San Francisco: Jossey- 

Bass. 

Merriam, S, & Simpson, E. (2000). A guide to research for educators and trainers of adults. (3rd edition). 

Malabar, FL: Krieger Publishing Company. 

Mezirow, J. (1995). Transformation theory of adult learning. In M. Welton (Ed.), In defense of the lifeworld: 

Critical perspectives on adult learning. Albany, NY: State University of New York Press. 39-70. 

Murphy, K., & Cifuentes, L. (2001). Using web tools, collaborating, and learning online. TechTrends; 45 (1), 28. 

126

Nesbit, T. (2001). Extending the boundaries: Graduate education for nontraditional learners. Journal of 

Continuing Higher Education, 49 (1), 2-10. 

Northrup, P. (2002). Online learners' preferences for interaction. Quarterly Review of Distance Education, 3 (2), 

219-26. 

Saltiel, I., & Russo, C. (2001). Cohort programming and learning: Improving educational experiences for adult 

learners. Malabar, FL: Krieger. 

Schrumm, L. (1998). On-line education: A study of emerging pedagogy. In Adult learning and the internet. 

New Directions for Adult and Continuing Education, no. 78. San Francisco: Jossey-Bass. 

Schwandt, T. (2000). Three epistemological stances for qualitative inquiry: Interpretivism, hermeneutics, and 

social constructionism. In N. Denzin & Y. Lincoln (Eds), Handbook of qualitative research (2nd ed), 189-214. 

Thousand Oakes, CA: Sage. 

Scribner, J., & Donaldson, J. (2001). The dynamics of group learning in a cohort: From nonlearning to 

transformative learning. Educational Administration Quarterly, 37 (5), 605-36. 

Sherron, G and Boettcher, J (1997). Distance learning: The shift to interactivity. CAUSE Professional Paper 

Series # 17. Retrieved September 19, 2001 from http://www.educause.edu/ir/library/pdf/PUB3017.pdf 

Strohschen, G. & Heaney, T. (2000). This isn't Kansas anymore, Toto: Team teaching online . In M. J. Eisen 

& E. Tisdell (Eds.), Team teaching and learning in adult education (pp. 33-42). New Directions for Adult and 

Continuing Education , no. 87. San Francisco: Jossey-Bass. 

Tu, C, & McIsaac, M. (2002). The relationship of social presence and interaction in online classes. American 

Journal of Distance Education, 16 (3), 131-150. 

Yin, R.. (1994). Case study research. Thousand Oaks, CA: Sage Publications. 

127

Corich, S. (2004). Book review: Instructional Design in the Real World: A View from the Trenches (Ann-Marie Armstrong). 



(Book Review) 

Reviewer: 

Stephen Corich 

Principal Lecturer 

Eastern Institute of Technology Hawke’s Bay 

Taradale, Napier 

New Zealand 

scorich@eit.ac.nz 

Textbook Details: 


Ann-Marie Armstrong 

US Government Printing Office, USA 

Information Science Publishing 

ISBN 1-59140-183-6 

This book is an anthology of thirteen articles examining the subject of instructional design from the viewpoint of 

real-world practitioners. The book provides an insight into the successes and failures of commercial instructional 

design projects, and gives practical advice to those who are interested in designing courses for commercial 

clients. 

The book is loosely structured around the ADDIE (Analyse, Design, Develop, Implement and Evaluate) 

instructional design process. Relevant articles that include business world case studies, learning theories, systems 

theory, and management theories and practices in a variety of commercial environments illustrate the steps 

within the process. The book concludes with a section entitled New Methodologies and System Integration, 

which deals with new ideas and methodologies and provides additional resources, strategies and lessons for real 

world instructional design. 

Analysis 

The first chapter “Concern Matrix: Analyzing Learners’ Needs” by Dr James A. Pershing and Hee Kap Lee, 

addresses the traditional learner analysis stage of the ADDIE process. They emphasize the need for active 

communication with the learner and introduce a learner analysis matrix that incorporates learner’s levels of 

concerns and perceptions. 

Chapter II, “Responding to the Learner: Instructional Design of Custom Built E-Learning” by Nick Carrick an 

instructional design and e-learning consultant, builds on the learner needs theme and presents a case study 

concerning the identification of learning aids for a hi-tech microchip manufacturing company. The case study 

reveals that the true value of instructional design lies in a learner-centered approach that prioritizes ease of use 

and learner control. 

Design, Development and Implementation 

Elizabeth Hanlis from the University of Alberta in her article entitled “Application of an Instructional Design 

Model for Industry Training: From Theory to Practice”, examines the problems encountered when attempting to 

apply an instructional design model. Two case studies, one involving a health company and the other a chemical 

company, provide examples of the need for prompt and effective action within a development environment 

dominated by limited budgets and high client expectations. 

Chapter IV, “Cultural Wisdom and Hindsight: Instructional Design and Delivery on the Run” by Jillian Rickett a 

technical designer from Sydney continues the design and develop theme, discussing the need for flexibility and 

adaptability in a cross-cultural environment. 






128

John Cox and Terry Armstrong, in their article “Combining Technology, Theory and Practice: A Finnish 

Experience”, stay with the design and develop theme when they describe their experiences of providing an 

undergraduate Strategic Management course in a remote Finnish location. The success of this case study relied 

on an approach, which integrated the Internet in both a traditional classroom setting and a remote location 

setting. 

In chapter VI, “Applying Contextual Design to Educational Software Development”, Mark Notess considers 

how contextual design can be applied to educational software development. The case study describes the design 

of an online tool for music listening and analysis. 

The final design, develop and implement chapter is written by the books author Ann-Marie Armstrong. In her 

article “What You See is All That You Get! A Practical Guide to Incorporating Cognitive Strategies into the 

Design of Electronic Instruction”, compliments that previous chapter and describes how cognitive strategies can 

be incorporated into the electronic learning environment. 

Evaluation 

There are three chapters dedicated to the evaluation phase of the ADDIE process. In the first, a group from the 

Belgium Center for Instructional Psychology and Technology, present a case study based on the evaluation of 

KABISA a computer based training program. Their article “KABISA: Evaluation of an Open Learning 

Environment” suggests that a systematic approach to instructional design remains highly valuable. 

The second evaluation chapter, “Guerilla Evaluation: Adapting to the Terrain and Situation” by Tad 

Waddington, Bruce Aaron and Rachael Sheldrick considers the impact of shifting focus from a training for 

activity approach to a training for results approach. It presents a model that links training to business goals while 

supporting continuous performance improvement. 

The final evaluation chapter “Standards for Online Courses: Can We Do It? Yes We Can!” by Noel Estabrook 

and Peter Arashiro of Michigan Virtual University, proposes various standards to be used for evaluating 

instruction. The chapter establishes the need for standards, describes the standards and explains how the 

standards can be used to effectively evaluate on-line courses. 

Methodologies and system integration 

The final three chapters introduce reusable learning objects and propose modifications to the ADDIE process 

model. The first chapter of the final section, written by Pam Northrup, Karen Rasmussen and David Dawson of 

the University of West Florida, introduces the concepts of Reusable learning Objects and Reusable Information 

Objects as tools that could facilitate quick and, systematic design and development of Web-based instructional 

materials. 

Chapter X11, “Integrating ICT in Universities: Some Actual Problems and Solutions” by Vassilios Dagdilelis of 

the University of Macedonia presents some of the problems arising when Information Communications 

Technology systems are integrated into universities and provides suggestions of how to deal with the problems. 

The final chapter of the book, “Integrated Training Requires Integrated Design and Business Models” by Arthur 

B. Jeffery from the Naval Education and Training Command and Mary F. Bratton-Jeffery from the University of 

South Alabama, introduces an implementation strategy called Quality Function Development. They endorse the 

feelings of the previous authors by suggesting that the ADDIE process should be expanded rather than replaced. 

Overall the book successfully uses the experiences of instructional design practitioners to support the 

requirement for a methodical approach to the design, development and implementation of instructional systems. 

The real-world case studies provide examples of how theory can be applied in practice and the tips and 

suggestions provide a valuable guide to all those involved in instructional system design. 

The author has done an excellent job in bringing together the experiences of a wide variety of instructional 

designers and presenting a cohesive book that covers all aspects of the instructional design process. The book 

offers an excellent reference to students, researchers and practitioners in terms of the application of instructional 

design processes, learning and management theories and communication tools and practices in a wide variety of 

different environments. 

129

Lee, Y. (2004). Software review: Review of the Tools for the Cognitive Task Analysis. Educational Technology & Society, 

7 (1), 130-139. 

Review of the Tools for the Cognitive Task Analysis 

(Software review) 

Reviewer: 

Youngmin Lee 

Department of Educational Psychology and Learning Systems 

Florida State University 

Florida, USA 

Tel: +1-850-575-1292 

yyl5185@garnet.acns.fsu.edu 

1. Introduction 

A cognitive task analysis is to model the cognitive process that a learner takes on when he/she performs certain 

task (Jonassen, Tessmer, & Hannum, 1999). That is, the cognitive task analysis is an aid to identify and analyze 

cognitive processes that underline performance of tasks in consideration of observable behavior. A lot of studies 

were conducted to implement effective and efficient cognitive task analysis method (Carlisle, 1986). For 

example, knowledge audit method provides the results of survey for clarifying a task through probing concrete 

examples in real context. In addition, task diagram method provides broad overview of a task and highlights 

unknown portion of cognitive process by asking an expert to break down the task (Uden & Willis, 2001). 

However, a computerized aid for a cognitive task analysis has been greatly paid attention because a cognitive 

task analysis is intrinsically labor intensive, time consuming, and costly activity. As a result, some computerized 

cognitive task analysis tools were developed and used to support that processes. Williams and Voigt (2000) 

reported that a computerized aid called CAT (Cognitive Analysis Tool) was very helpful to conduct a cognitive 

task analysis in terms of accuracy and consistency of the work. The functions of the tool are goal setting, method 

selection, condition selection, undefining step, subgoal step setting, and primitive step setting. Following a linear 

process, a user can create detail description of each task and condition. In addition, he/she can add and link a 

goal that he/she specified and method, and condition. 

Welie, Veer, and Eliens (1998) developed a cognitive task analysis tool named Euterpe which focuses on 

hierarchical structure of tasks and relationship between the tasks and their properties. Basic concept underlying 

this tool is that a task is composed of object and a variety of properties and the task analysis is to connect them 

by specific rules. Another tool, HTA (Hierarchical Task Analysis) tool graphically illustrates split tasks and 

imposes a purpose, responsibility and condition of those tasks in hierarchal task branches. Microsaint was also 

developed to assist and facilitate a cognitive task analysis. Those tools listed above visually display a task and 

the flow of task analysis process by dragging and dropping a function. Especially, Microsaint adopts the 

simulation techniques in conjunction with the cognitive task analysis process. 

The purpose of this study is to review the theoretical background and functions of each tool and, as a result, 

suggest some findings for developing new cognitive task analysis tools. Especially, this study seeks to clarify 

following questions: 1) What theory is being adopted in the tools; 2) What methodology is being used in the 

tools; 3) What functions do the tools have; and 4) What implications the study findings have on the following 

new tools for the cognitive task analysis. 

2. Review of the tools 

A. Euterpe 

Product details: 

Product name: Euterpe 

Product Category Cognitive task analysis tool 

Producer name: Welie, M., Veer, G. C., & Eliëns, A. 

Product link: http://www.cs.vu.nl/~gerrit/gta/euterpe.html 






130

Snapshot review: 

Ease of installation **** 

Ease of navigation *** 

Documentation * 

Pedagogical Foundation ***** 

Instructional Value **** 

Production Value ** 

Interactivity *** 

1. Overview 

Euterpe was developed on the basis of an object-oriented programming approach. It disseminates the task into 

several object components such as object, attributes, actions, role, and events. Each component can be defined by 

its own characteristics. The user manipulates these components and makes a relationship among them by simply 

drawing a line that connects them. Euterpe supports a linear-fashioned task analysis process in series. 

Basically, Euterpe uses the Group Task Analysis (GTA) approach, which refers to the task as the combination of 

the situation, the people, their roles, and the work. This concept is reflected in the functions of Euterpe. In the 

past, task analysis has focused mainly on analyzing a single user and his/her tasks. Groupware Task Analysis 

expands task analysis by looking at a task-world from the perspectives of work, agents, and situation. 

It regards the task-world as an organization where many people do tasks, work together and interact with both 

people and objects. From the perspective of work, GTA looks at tasks, goals, actions, and procedures within an 

organization. On the other hand, from the perspective of agents, GTA looks at the people and machines that 

perform the work, what their role is in the organization, and how responsibilities are allocated. 

2. Features 

In the menu bar, there are ‘File’, ‘Edit’, ‘Insert’, ‘View’, ‘Window’, and ‘Help’ buttons. The File menu is used to 

open, save, print, and export the file. The Edit menu is used to cut, copy, and delete the task template. The Insert 

menu is used to put in the task temple in a row or series. The View menu is used to insert the task template and 

define specific constraints. The Window menu is used to arrange the icon and windows. The Help menu is used 

to enable the users to find the topics by index. 

131

A template can be used to specify the details and name of a task. Select the View Template from the View menu 

or double click on the node in icon bar. A new window will pop up showing many details of the task. The user 

can fill in all the appropriate details and close the window by choosing ‘Ok’ or ‘Cancel’. 

The user can insert either child nodes or sibling nodes, which stand for a unit of the task. The corresponding 

function can be selected through the ‘Insert’ menu in menu bar by using the ‘Ins’ key, or by selecting the 

function from the context menu (right mouse button). The user can delete a single node or delete a whole subtree 

hanging on a node. Select the ‘Delete’ from the Edit menu to delete a node or Delete subtree to delete the whole 

subtree. 

In the Icon Bar, there are ‘New’, ‘Open’, ‘Save’, ‘Cut’, ‘Copy’, ‘Paste’, ‘Do’, ‘Undo’, ‘Add’ the task template in 

a row or series, ‘View’ percentage window, ‘Print’, ‘About Euterpe’, and ‘Help’ icons. 

There are five taps below the icon bar. The ‘Task’ tap is to show the flowchart of analysis results. The ‘Object’ 

tap stands for physical entity, name and value pairs. The ‘Agent’ is a class of individuals with certain 

characteristics. The ‘Role’ tab is a meaningful collection of tasks performed by the agent. The ‘Task’ tab is the 

actions to be performed to reach a goal. The ‘Event tab’ tap is the change in the state of the task. 

Each tap has its own template that is used for defining the properties of the task. Therefore, there are ‘Task’ 

template, ‘Object’ template, ‘Agent’ template, ‘Role’ template, and ‘Event’ template. Each template usually 

consists of name, type category which is ‘Individual’, ‘Organization’, ‘Role’, ‘Media support’, ‘add and 

remove’, and specification of its characteristics. 

3. Implications 

The task analysis process can be represented graphically. All tasks are represented in boxes and lines. This 

representation enables the users to organize the hierarchy and procedures of the task and task element. However, 

the types of representation are so limited that other representation systems, such as a box, a triangle, a circle, and 

a rhombus should be considered to present the types of task more effectively. 

The task itself can be divided into the task elements, properties, and commentary. The task element is a unit of 

task. The task element has several properties, which include the characteristics and conditions. The commentary 

is needed to provide the descriptions of the relationship between the task elements and the properties. 

The task can be represented in multiple ways. The existing tools such as Euterpe, basically assume that all tasks 

can be represented in various ways if represented by multimedia objects, such as a graphic, sound, and movie 

file. The user should have to represent the task with multimedia types by manipulating some functions which are 

provided by the tools. 

The complex level of the task should be considered. Euterpe emphasizes complex tasks and their linkages. The 

user can’t determine the level of complexity when he/she analyzes the job and the task. By adding complexity 

level icon and/or designating the complex level when starting the analysis, the user can represent the task more 

specifically. 

B. HTA (Hierarchical Task Analysis) 


Product name: HTA (Hierarchical Task Analysis) 


Producer name: Humanreliability associates 

Product link: http://www.humanreliability.com/software.html 


Ease of installation **** 

Ease of navigation **** 

Documentation ** 

Pedagogical Foundation ** 

132

Instructional Value *** 

Production Value ** 

Interactivity ** 

1. Overview 

Hierarchical Task Analysis (HTA) tool is a systematic support tool for describing how tasks are organized. It 

identifies the overall objective of a job, the overall goal of the task, what sub-tasks are needed to achieve that 

goal, and when each sub-task should be carried out. 

The users always start by stating the overall objective. Then this objective is broken down into sub-tasks. These 

tasks will have a certain order or set of conditions under which each sub task should be performed. This is called 

a ‘Plan’. 

The HTA adopts the hierarchical task analysis approach in which the users break down and display the task 

hierarchically from top to bottom to show a hierarchical relationship among the tasks. Thus the users easily 

understand the path of the task sequences and their relationships. 

2. Features 

In menu bar, there are ‘File’, ‘Edit’, ‘Insert’, ‘View’, and ‘Options’. The File menu is to open, save, and print the 

file and define the properties of the task and its procedures. The Edit menu is to cut, copy, and paste the task 

templates. The View menu is to select the page, which is previously ‘Split and Zoom’ function. The Options 

menu is to adjust the view of graphic interface. 

In the icon bar, from the left, there are ‘Split Window’, ‘Center Focusing’, ‘Go To Top Menu’, ‘Move Up Tree’, 

‘Insert Page Down’, ‘Insert Page Left’, ‘Insert Page Right’, ‘Child Box’, ‘Left Box’, ‘Right Box’, ‘Parent Box’, 

‘Preconditioned Box’, ‘Plan’, ‘Erase’, ‘Expand’, ‘Contract’, ‘Zoom In’, ‘Zoom Out’, and ‘Add a Unit’. 

Basically, HTA has four templates: ‘Properties’ template which specifies source file, project name, task goal, 

analysis status, date, and backup interval. ‘Procedure’ details template has the task goal, purpose, 

responsibilities, and pre-conditions. ‘Page Select and Reorder’ template is used to rearrange the order of the box 

and its labels and ‘Split-screen’ template is the description of each task number and piece of information. 

133


The HTA enables the user to sequence the logic of the task by adding child and parent nodes hierarchically. This 

type of cognitive task analysis tool may be more suitable to hierarchical tasks which emphasize intellectual 

skills. This gives the researcher an idea that the user should be allowed to manipulate the sequence of the task 

based on the type of task. That is, the user may need to organize the sequence of the task depending on the task 

type such as intellectual skill, motor skills, and verbal information. 

If the task is performed by several users collaboratively and/or has sub-tasks performed by different users, HTA 

indicates who performs the task. This can make it evident who is responsible for the task or sub-task (i.e., 

accountability). In addition, this can be basic data for tracking a specific user’s performance. 

There should be specific functions for merging some tasks. The more complex the task is, the better the task is 

congregated. If the complexity of the task is increasing, the user may not identify the entire task obviously. So, if 

the user can combine some tasks according to the job/task specification in one group, the user will recognize 

whole structures and sequences of the task at a glance. 

The characteristics of job and task should be more specified. When the user defines the job and task, he/she 

needs to describe detail information of the job and the task such as the job/task goal, purpose, constraints, and 

pre-defined conditions to be performed. In addition, these elements should not be separated differently. The 

commentary, which is the descriptions of the relationship between the task elements, can be an example. 

With specific characteristics of job and task, there should be specific criteria for achieving these job and tasks. 

This criteria or guidelines can help the evaluator as well as the learner to gain information about achievement 

level and minimum requirements. HTA seems to overlook the importance of evaluation of the tasks. These 

criteria or guidelines can be suggested as pull down menus based on performance, complexity, and acceptance 

level of the job and the task. 

C. CAT (Cognitive Analysis Tool) 


Product name: CAT (Cognitive Analysis Tool) 


Producer name: Kent E. Williams 

Product link: http://www.iems.ucf.edu/ver40/faculty/william.htm 


Ease of installation ***** 

Ease of navigation ** 

Documentation *** 

Pedagogical Foundation ***** 

Instructional Value *** 

Production Value *** 

Interactivity ** 

1. Overview 

The CAT is designed to support a cognitive task analysis. This tool begins by suggesting two options, which are 

‘Tutorial format’ and ‘Guidance format’. One of the options, the tutorial format is led by an electronic manual 

and its topics and the guidance format is led by step-by-step procedures for analyzing a task. 

The users should set the goal at the starting point and then specify the procedures in conjunction with stating the 

conditions. After finishing those processes, the user can add some description of them. 

This CAT was developed based on the GOMS (Goals, Operators, Methods, and Selection rules) model. A 

GOMS model is composed of Methods that are used to achieve specific Goals. The Methods are then composed 

of Operators at the lowest level. The Operators are specific steps that a user performs and are assigned a specific 

134

execution time. If a Goal can be achieved by more than one Method, then selection rules are used to determine 

the proper method. 

2. Features 

The system menu area consists of the ‘File’, ‘Edit’, ‘Navigate’, and ‘Help’ drop down menus located below the 

title bar area. The File menu contains the ‘File Manipulation’ options, ‘Print Capability’ options, and 

‘Application Termination’ options. The Edit menu contains the model edit options used. The Navigate drop 

down menu is used in conjunction with the graphical user interface to navigate through the model, to provide a 

method of returning to ‘Guidance Mode’, and to execute an existing model. 

The Help menu is to assist the users. For example, it presents the menu items such as ‘Tutorial’ which presents a 

complete tutorial for using the CAT tool, ‘Help’ which presents detailed help for using the CAT. Interestingly, 

this menu has the option of ‘How to Use Help’. This menu option presents the users a lot of information on how 

to use the help system. 

From the top to the down, first icon represents the ‘Goal’, which allows the user to open a new model. Second 

icon, ‘Method’ is to add a new method to the current goal structure. Third icon ‘Selection condition’ is to add a 

new selection rule condition node to the model. Fourth icon, ‘Undefined step’ is provided only as a visual cue to 

indicate that there is undefined step in model. 

Fifth icon, ‘Sub goal step’ is to add a new sub-goal node to the current goal structure. Sixth icon, ‘Primitive 

Node’ is to add a user defined or predefined primitive node to the current goal structure. Seventh icon, ‘Group 

Step’ is to place over a method step will group it with the next method step. Last icon, ‘Ungroup Step’ is to 

enable the user to ungroup a step from those following by clicking and dragging this icon to the step to be 

ungrouped. 

When a user starts CAT, he/she can choose ‘Tutorial’ button which provides the user to access to the CAT 

tutorial when detailed step by step information about developing models using CAT is required or ‘Guidance’ 

button, which provides the novice or intermediate user with a method to develop a ‘CAT’ model in a completely 

constrained ‘Guidance’ mode. 

The first input is to define the top level goal to be accomplished by the model. Once a top level or any other goal 

is defined, it is necessary to define the steps (sub-goals) for all the methods associated with it. ‘Steps’ are a 

135

sequence of operators associated with the goal which or may not be further decomposed into additional methods 

and steps. 

CAT provides the users several templates such as ‘Enter sub goal information’ template, which is to define a 

sub-goal or primitive operator in the tool, ‘Decision step information’, which is to define a condition and an 

action to take if the condition is true. 

When the entering of the steps for a method has been completed, it may be necessary to specify a change in the 

order in which the steps were entered or to specify that step order is unimportant. 


A new tool for the cognitive task analysis has to have a printing function which reports the user’s performance in 

terms of the characteristics of the job and the task. CAT has a function to report the results of cognitive task 

analysis. However, it does not seem to consider individual results of the performance in doing the task analysis. 

With drag and drop method, the user should be able to input the condition, goal, purpose, and method with text. 

Especially, it can be effective for the user to follow a pre-determined step-by-step analysis process at the 

beginning of the analysis. 

There is a specific function called ‘Grouping’ for merging some tasks. Unlike the HTA analysis tool, CAT 

enables the user can combine some tasks, conditions, and methods according to the job/task specification in one 

group. The user can recognize whole structures and sequences of the task at a glance. 

The condition clause (If-then) function can make the sequences of the task more flexible. For instance, if one 

task can be done in a different way, the if-then function allows the user to consider setting a different task 

sequence based on the user’s choice. 

D. Microsaint 


Product name: Microsaint 


Producer name: MA&D 

Product link: http://www.maad.com 


Ease of installation ***** 

Ease of navigation **** 

Documentation *** 

Pedagogical Foundation *** 

Instructional Value **** 

Production Value **** 

Interactivity ***** 

1. Overview 

Microsaint is a simulation-based task-analysis tool. Like other tools, this tool is used in different types of 

cognitive task analysis. However, this tool generates more powerful functions of the simulation to make the task 

more dynamic. 

Microsaint has strengths in specifying the descriptions of the task, designing the task flow, exchange the 

commands, and using the resource wizard. Unlike other tools, this tool does not provide any guidelines or rules 

when the users start the tool. The users can go to the main page directly and analyze the task. 

136

This tool can support hierarchical and procedural analysis of the task. A hierarchical task analysis is developed 

bottom up, from general to specific. A hierarchical task analysis is based on learning taxonomies, starting from 

the most complex. 

The nature of the terminal task determines at which level in the taxonomy one should start breaking down the 

task from more complex to less complex, going through each of the learning levels. 

A procedural task analysis is developed linearly and sequentially, that is, step-by-step procedure. It has a 

directional flow. It has a start and an end. A procedural task analysis is not concerned with the levels of the 

learning taxonomies, it is procedural in nature. 

If the task is a relational rule, then the steps of the task analysis would include how to apply this rule. If the task 

is concept learning, then the task analysis would include how to determine whether a particular instance is an 

example of this concept. 

2. Features 

In the Menu bar, the ‘File’ menu includes the ‘New’, ‘Open’, ‘Save’, and ‘Print’ menus. The ‘Edit’ menu 

includes ‘Cut’, ‘Copy’, ‘Paste’, ‘Delete’, ‘Select all’, and ‘Check Syntax’. The search menu contains ‘Find and 

Replace’. The ‘Display’ menu contains the network diagram, variable catalog, function library, event queue, 

snapshot, action view, resource wizards, and external model calls. 

The ‘Execute’ menu includes the ‘Go’, ‘Single’ step, and ‘Normal speed’ etc. The ‘Action View’ menu is to 

adjust the speed of media player. The ‘Analyze’ menu includes the result of statistics, various graphs, and the 

scales. The ‘Option’ includes the references. The ‘Window’ menu is to change the arrangement and the size of 

the windows. The ‘Help’ menu includes the help and using the help. 

From the left side, the first icon bar shows ‘New’, ‘Open’, ‘Save’, ‘Print’, ‘Print Preview’, ‘Select All’, ‘Cut’, 

‘Copy’, ‘Paste’, ‘Find’, ‘Replace’, and ‘Help’. In second icon bar, from the left side, this icon bar shows 

‘Network Diagram’, ‘Variable Catalog’, ‘Function Library’, ‘Event Queue’, ‘Snap Shot’, ‘Execution Monitor’, 

‘Action View’, ‘Optimize, Resource Wizard’, ‘Setting’, ‘Check Syntax’, ‘Go’, ‘Pause’, ‘Single Step’, ‘Top 

Speed’, ‘Add new item to list’, and ‘Halt’. 

137

In third icon bar, this icon bar shows ‘Select’, ‘Network’, ‘Task, Queue’, ‘Path’, ‘Delete Path’, ‘Start’, ‘Text’, 

‘Near’, ‘Far’, ‘Undo’, ‘Down’, ‘Go To’, ‘Up’, ‘Create’, ‘Move’, ‘Delete Icon’, ‘Vertical Axis’, ‘Horizontal 

Axis’, and ‘Remove Graph’. 

Microsaint has several templates such as ‘Network Description’ template, which includes the network box 

selection, network number, and notes, ‘Task Description’ template, which includes the task box selection, task 

number, time, condition, beginning and ending effects, and ‘Job Description’ template, which includes the job 

queue number, queue number, entering effect, priority, and departing effect. 

The ‘Resource Wizard’ template is used to create or select a resource variable and to allocate resources to the 

tasks in a model. To display the Resource Wizard, from the Display menu, the user can choose Resource Wizard 

or click the Resource Wizard button on the toolbar. The user can also click the Resource wizard button in the 

‘Variable Catalog’ dialog box to open the Resource Wizard and follow the instructions in the Resource Wizard. 

The user can quit the Wizard at any point by clicking ‘Cancel’. 


Tasks can be represented as multimedia elements. The Microsaint supports the user in representing the tasks in 

various ways such as a graphic, sound, and movie file. In addition, the user can manipulate specific properties of 

the multimedia elements. For instance, the user can adjust the starting point, duration, ending point, effects, and 

the play of the element by clicking the icon. 

The task library can be considered as a database. The user can select a specific task or task example in the 

library. The user attaches this task and task property in current designed task analysis template. The task library 

can be organized by type of job and task, knowledge and skills, performer, and complexity of job and task. 

The result of performance can be sorted and stored within the user’s personal database. If so, the new system for 

the cognitive task analysis has the criteria for each task if it is achieved. In addition, a personal database should 

be developed to sort and distribute each performed task by performance criteria and types of job and tasks. 

The resource wizard can be usefully used to create or select a task and task-process elements and to allocate 

these elements according to the tasks. Also, the wizard can be useful to start initial task analysis (i.e. Draft 

version of task analysis). If the resource wizard were successfully designed and implemented, only thing the user 

to do is to follow each guidelines suggested by wizard. This can make the task analysis process effective as well 

as efficient. 

3. Conclusions 

The reviewer summarized the snapshot of the tools. This can be helpful to compare some features of the tools. 

Euterpe HTA CAT Microsaint 

Ease of installation **** **** ***** ***** 

Ease of navigation *** **** ** **** 

Documentation * ** *** *** 

Pedagogical Foundation ***** ** ***** *** 

Instructional Value **** *** *** **** 

Production Value ** ** *** **** 

Interactivity *** ** ** ***** 

This study implicates 1) Existing tools should be redesigned to supplement their own weakness of functions and 

support the users to do the cognitive task analysis more flexible; 2) These tools should allow the users to choose 

one of the cognitive task analysis methods; 3) Some tools are based on DOS, not Windows. Various options for 

installing the tools should be considered; 4) New types of a cognitive task analysis tool can be considered to 

support cognitive task analysis in light of performance effectiveness, learnability, error tolerance, interactivity, 

user guidance, and screen layout; 5) Cognitive task analysis tool may be needed to connect with a job task 

analysis tool, competency analysis tool, or human performance tracking system. 

138

References 

Carlisle, K. (1986). Analyzing Jobs and Tasks. Englewood Cliffs, NJ: Educational Technology Publications. 

Jonassen, D.H., Tessmer, M. and Hannum, W.H. (1999). Task Analysis Methods for Instructional Design. 

Mahwah, NJ: Lawrence Erlbaum Associates. 

Uden & Willis, M. (2001). Designing User Interfaces using Activity Theory. Thirty-fourth Hawaii International 

Conference on System Sciences. (HICSS-33). Wailea, Maui, Hawaii, USA -Software Process Improvement. 

IEEE Computer Society Press. 

Welie, M., Veer, G. C., and Eliens, A. (1998). Euterpe - Tool support for analyzing cooperative environments. 

Ninth European Conference on Cognitive Ergonomics, Limerick, Ireland. 

Williams, K. E. and Voigt, R. E. (2000). Evaluating a Computerized Aid for conducting a Cognitive Task 

Analysis. In Proceedings of the 2000 International Conference on Industry, Engineering, and Management 

Systems. Cocoa Beach, Florida, March 13-15. 

139

International Forum of Educational Technology & Society 

http://ifets.ieee.org/periodical

Complete issue in PDF - Educational Technology & Society

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?