January 2012 Volume 15 Number 1 - Educational Technology ...

Table 5. Questions that guided revision of learning console
Criteria Questions asked
Validity Is the product an innovative element to support the Malaysian online learners?
Is the product consistent with the design objectives?
Practicality Does the product run as intended?
Can the intended learners actually use the program?
Does the product appeal to the users?
Effectiveness
User friendliness, free
from errors, efficiency,
ease of use
Does the product provide opportunities for meaningful learner-instructor interaction?
Does the product motivate learners?
Does the product assist the learners in directing their own learning?
Does the product support them in their learning process?
Is navigation clear to users?
Are the meanings of icons clear?
Do users get lost in navigating through the program?
Is the product efficiently constructed?
What is the minimal technology that it can be run on and do the target
learners have access to that level of technology?
Formative evaluation of the learning console
The researcher conducted a qualitative evaluation during which the learning console was subjected to scrutiny by 40
research participants (experts, instructors, and learners) who were observed, consulted, and interviewed at four
phases of the evaluation. The phases as promulgated by Flagg (1990) were (1) the planning and needs assessment
phase; (2) the pre-production formative evaluation, which referred to the collection of information to guide decisions
during the design phase; (3) the production formative evaluation, which referred to the gathering of data to guide
revisions using the prototype; and (4) the implementation formative evaluation, which referred to testing the
effectiveness of the product/program under normal use conditions with the intention of still making changes. The
needs assessment phase was regarded as the first phase of evaluation by Flagg (1990), where pertinent information
such as the learner profile was gathered.
Table 6 summarizes the different phases of the evaluation and the evaluation techniques associated with each phase.
Table 6. Summary of the evaluation phases and evaluation techniques involved
Phases of research Phases of formative evaluation Evaluation technique
Analysis Planning and needs assessment Situational analysis (past research and document
analysis)
Learner analysis
Learning environment analysis
Design Pre-production evaluation Interviews with learners and instructor
Concept walkthrough — analysis of group discussions
Development Production evaluation Peer review
Expert reviews
One-to-one evaluation (learners and instructor)
Implementation Implementation evaluation In-depth interviews with learners and instructor
Survey
Online observation of students using the learning console
Expert consultation
Findings and discussion
The peer reviewer and the four experts generally agreed on the potential of the learning console to provide guidance
and regulate motivation, to enhance self-evaluation, and to promote interaction and knowledge building among
learners and the instructor. The instructors saw the learning console as an option to the existing mode of assessment
147