Education for a Digital World Advice, Guidelines and Effective Practice from Around Globe, 2008a

9 – Quality Assurance by Design
all stakeholders’ benefits. There are identification of key
variances and dynamic evaluation.
Identification of key variances: All organizations
need to function well without problems. The weakest
links should be identified, eliminated, or at least controlled.
Working on socio-technical design, Mumford
(1983) believed that design needs to identify problems
that are endemic to the objectives and tasks of organizations.
Intentional variances stem from the organizational
purposes and targets. Operational variances predate
design, and are the areas the organization has to
target. They stem from the operational inadequacies of
the old system, and the technical and procedural problems
have been built into it inadvertently. “Key variances
refer to the same variance in both intentional and
operational levels”.
Design and engineering are connected to both intentional
(pedagogical) and operational (engineering) approaches.
Sometimes there are problems, called variances
in socio-technical design. From an educational perspective,
Schwier and his colleagues (2006) emphasized the
need of intentional (principles or values) and operational
approaches (practical implications), and provided
an analytical framework of the gaps and discrepancies
that instructional designers need to deal with. The identification
of a key variance helps the organization to
provide added-value outcomes. This is achieved by the
use of dynamic evaluation.
Dynamic evaluation: According to Lambropoulos
(2006), e-learning evaluation aims to control and provide
feedback for decision-making and improvement. It
has four characteristics: real-time measurements, formative
and summative evaluation, and interdisciplinary
research. Dynamic evaluation links and informs design.
It also provides immediate evaluation to user interface
designers. In addition, it identifies signposts for benchmarking,
which makes comparisons between past and
present quality indicators feasible (Oliver, 2005). Such
dynamic evaluations will enable the evolution of design
methods and conceptual developments. The use of several
combined methodologies are necessary in online
environments. Andrews and colleagues (2003), De Souza
and Preece (2004), and Laghos and Zaphiris (2005) are
advocates of multilevel research in online, and e-learning
environments. Widrick, cited by Parker, claimed that:
“[it] … has long been understood in organizations that
when you want to improve something, you first must
measure it” (2002 p. 130). Parker (2003 p. 388), does not
see that engineering for unified learning environments is
feasible:
“The engineering (or re-engineering) of systems
designed to guarantee that manufacturing processes
would meet technical specification might
seem to imply a uniformity that may not be possible,
or even desirable, in the dynamic and heterogeneous
environment of higher education.”
According to Parker, a unified systems design is not
possible, or even desirable. The interdisciplinary nature
of e-learning, the large number of stakeholders involved,
and the uniqueness of the context make e-learning engineering
extremely difficult. Nichol and Watson (2003, p.
2) have made a similar observation: “Rarely in the history
of education has so much been spent by so many
for so long, with so little to show for the blood, sweat
and tears expended”. 14 It is contended that e-learning
engineering, including dynamic evaluation, may well
minimize the cost. User interface designers should recognise
the need to limit this process to a period of days
or even hours, and still obtain the relevant data needed
to influence a re-design (Shneiderman & Plaisant, 2005).
At present, the design process is still vulnerable to the
Hawthorn effect (Faulkner, 2000). Laboratory research
ignores the distractions of e-learner behaviour in the
real world. On the other hand, dynamic evaluation enables
the evolution of design methods and conceptual
developments (Silius & Tervakari, 2003; Rogers, 2004).
Ethnography captures events as they occur in real life,
and then uses them for design. It can be a time-based
methodology aiming for a description of a process in
order to understand the situation and its context, and to
provide descriptions of individuals and their tasks (Anderson,
1996). This type of research could be said to be
part of dynamic evaluation in e-learning engineering
(Figure 9.3):
Figure 9.3 Formative and summative evaluation in e-learning communities
The line from A to B in Figure 9.3 represents the lifespan
of an e-learning community. A short or long term
e-learning community may have a beginning (A) that is
the baseline, and an end (B). Usually, the comparison of
14
Editorial “Rhetoric and Reality—The Present and Future
of ICT in Education” for the British Journal of Educational
Technology, by Nichol and Watson (2003:2).
Education for a Digital World 121