DICE Project Final Report Resource Discovery Tools Evaluation and ...

User Evaluation
DICE Final Report
The feedback from the User Evaluation is summarized here and recorded in full in Appendix B.
We make some comment first on the nature and conduct of the evaluation.
While the focus of the exercise was to evaluate D+ (in the context of other available discovery
tools and services), we were also looking for feedback in related areas (integration, re-use of
resource lists, educational focus). For users who were not especially familiar with resource
discovery or even e-learning, this was quite challenging, so evaluators were provided with a
briefing paper introducing Resource Discovery and explaining ‘Advanced Topics’. Our Resource
Discovery Tools Guide (Appendix A) included many screenshots using different RD tools and
services to make it more easily readable, and a Quick Guide was also produced, aimed at naïve
users. Descriptions of the different RD Tools drew attention to particular features relevant to the
evaluation. The on-line feedback form was also designed to be user-friendly, consisting of only a
dozen or so open questions encouraging free-form reply.
While we were only partially successful in capturing information on all points, the evaluation was
nevertheless useful, endorsing our own findings and bringing to light other issues. It is one thing
to be aware of a particular issue, quite another to decide on how to act on that in light of the
feedback.
It proved more difficult to recruit evaluators than anticipated, especially outside the university. We
advertised on several mailing lists amongst e-learning practitioners, locally, regionally and
nationally; made presentations to VLE user groups in the West Midlands; contacted colleagues in
the SURF colleges and posted flyers at both our Stafford and Stoke campus. Ultimately though,
only about half of those who were engaged or expressed an interest responded. We had
allocated a certain budget to remunerate evaluators, which was just as well: we would not
recommend anyone should rely on goodwill to conduct such an exercise.
There is ample evidence from our own and others’ research of interest, if not practice, in this
area 6 . Contrary to our initial expectation (as described in our RD Tools Guide) several
respondents from the FE sector indicated that they thought D+ was more appropriate for HE, and
that FE colleges might have other issues in implementing such. Thus, locally at least, the
evaluation highlighted significant cultural differences between HE and FE relating to the
curriculum, infrastructure (e.g. firewalls) and e-learning practice (e.g. authentication).
Certain input to the evaluation resulted from follow up conversations and email correspondence,
outside of the on-line submissions.
Feedback Summary
The feedback reflected the different expectations of more less experienced users. We were
grateful to receive input from several practitioners who had particular expertise in resource
discovery, but other respondents seemed not to have read the Guide or else were less than
conscientious in their replies. Evaluators were recruited from students, education technologists,
academics, and other e-learning practitioners (managers) mainly at local FE and HE institutions
as follows
academics 3
students 5
education technologists 4
practitioner / mgr 3 FE / HE 4 / 11 Non returns 8
6 Considering anecdotal evidence, content and trends in VLE implementation (BB building blocks,
repositories), commercial systems such as Sentient, Discover, the CREE project, related work at
Edinburgh, Heriot Watt, Manchester, other HE institutions.
11