October 2007 Volume 10 Number 4 - Educational Technology ...

The final incremental prototyping step was to install IMMEDIATE’s component applications on a LAN to ensure
that they would work correctly together to support the key learning computer requirements. Once we had achieved
this our next task was to evaluate these features under field conditions.
Evaluation with users
IMMEDIATE was installed to run over the rural telephone network and then tested with volunteer users from a
remote farming and fishing community over a period of two days. This involved setting up the Learning Shell on a
PC, the selection of four volunteers, a pilot study with one user and an in-depth study with the other three. If
IMMEDIATE was to have universal application then it was vital to test it under some of the more difficult field
conditions that might be faced by distance students. It was necessary to ensure that the essential functionality of the
system was accessible with an older model PC and operating system, a slow and unreliable rural Internet connection,
and a relatively inexperienced computer user working alone.
The goals of this evaluation were to determine that IMMEDIATE performed reliably under these conditions and that
all of the learning computer's functionality was accessible to the user, and to assess the usability of the Learning
Shell. Developers of any computer-based learning must consider students both as learners and as users, that is, they
must design both for form and content (Smulders, 2003). Poor usability inhibits learning. Inaccessibility prevents it
altogether. This is a critical issue where the student is studying at home alone without access to user support.
In this phase of the project, the evaluation was focussed on the student as user, that is, on the technical question of
whether the users could carry out the main tasks, without considering the pedagogical effectiveness of the learning
computer. This involved evaluating the functionality of all three subsystems – the Repository Manager, the Course
Authoring and Management System and the Learning Shell. For students to carry out the specified tasks, all the three
subsystems had to work together effectively.
The usability of the student end also had to be evaluated. Could the participants complete a set of typical learning
tasks unaided? How efficiently could they do this?
Finally, the following questions needed to be addressed with regard to accessibility. Was the system runnable where
the Internet was slow and unreliable? Could key functions, such as updating resources, communicating with other
students and getting help from the tutor, work over a rural network?
The Learning Shell was installed on PC running Windows 95 at a small rural school. Four members of the
community serviced by the school volunteered to assist in evaluating the usability of the prototype. All four
volunteers had studied at the tertiary level and three were current or very recent tertiary distance students. One
agreed to make herself available for a pilot study which was used to refine the evaluation to ensure that all major
aspects of the system would be tested.
The three remaining volunteers participated in the in-depth study. They were selected on the basis that they lived in a
rural area, had distance learning experience and some knowledge of computers (the relevant information was
obtained by a questionnaire). This is an example of purposive sampling (Yin, 1984, Patton, 1990) where appropriate
individuals who meet the specified requirements are selected. It was necessary to find individuals who had the
potential to successfully complete the exercises under conditions similar to those faced by a distance student,
working unassisted from a remote location.
The three volunteers were asked to complete seven scenarios covering setting up the student Learning Shell and
exploring all aspects of its functionality, including accessing learning material in six different study modes: lecture,
group work, tutorial, textbook, collaboration, and practice. They were expected to complete these in two one-hour
sessions spread over two days. The first exercises were quite detailed in their instructions, the later ones,
progressively less so. To ensure that no user was advantaged by knowledge of the course content, the material was
taken from a third year university paper on a topic that none of them had studied. This was appropriate given the
nature of the evaluation. The sequence of scenarios was as follows:
1. Initialise Learning Shell to own preferences
2. Log onto course
149