10. Briefing Paper Template - Higher Education Academy

Briefing Paper 

Technology Enhanced Assessment for Learning: Case Studies and 

Best Practice 

Briefing Paper by: John Dermo, University of Bradford (March 2011) 

Overview 

This seminar explored how instructors on foundation degree and first year 

undergraduate courses can deliver formative e-assessments, giving automatic generic 

feedback to assist and enhance their students’ learning 

1. Abstract: please provide a brief abstract of the seminar delivered (maximum 200 words). 

The seminar first looked at the literature of assessment in general as well as technology 

enhanced assessment, to set the background for two case studies. These two case studies, 

from the fields of Clinical Sciences and Engineering, provided varied practical examples of 

how formative e-assessment is being carried out in two different subject areas at the 

University of Bradford. This was followed by a group discussion of the major issues related 

to formative electronic feedback, and delegates were able to share experiences with other 

practitioners and to identify the main challenges facing formative e-assessment, especially 

in the HE sector. In addition, delegates were given a guided tour of the University of 

Bradford’s dedicated e-assessment suite, which is used both for summative and formative 

technology enhanced assessment. 

2. Rationale: please provide the background context, such as the research/evidence-informed 

practice context, which provided the impetus for the seminar. 

In recent years there has been a remarkable increase in the use of e-assessment in the 

higher and further education sectors. Much of this, though, has focused on summative 

assessment, often primarily driven by administrative efficiency and to save time for busy 

academics teaching on large undergraduate modules. However, in the field of ELTT 

(enhancing learning and teaching through technology) there is now less discussion of 

summative assessment - “e-assessment of learning” - and more interest in how 

technology-enhanced assessment can benefit the learning process through formative 

feedback: this is “e-assessment for learning”. This is partly in response to a clear demand 

from students (eg through the National Student Survey) for increased quality and quantity 

of feedback. 

This seminar sought to provide examples of formative assessment which reach beyond the 

stereotypical cliché that e-assessment can only be used for multiple choice questions to 

test knowledge, and to challenge the oft-heard accusation that reliance on e-assessment 

can lead to “dumbing down” of learning. The aim was to investigate whether it is, in fact, 

possible to use generic automated feedback which can genuinely be called “assessment for 

learning” and might enhance the learning experience for large numbers of university 

students in a practical way. By using a range of question types and careful question, 

EvidenceNet is a Higher Education Academy resource. 

www.heacademy.ac.uk/evidencenet 

Page 1 of 9


feedback and learning design, the case studies demonstrated how to assess learners at 

higher levels (Bloom, 1956). 

The seminar built on the achievements of the University of Bradford’s HEA e-Learning 

Pathfinder Project “Embedding support processes for e-assessment” and Bradford’s JISC 

Institutional Exemplar e-Assessment Project, “Integrating thin client systems for secure e- 

assessment” (ITS4SEA). 

The case studies were closely aligned with HEFCE’s framework in “Enhancing Learning and 

Teaching through the Use of Technology” (HEFCE, 2009), in particular with the strategic 

priorities related to innovation in teaching and learning, enhancing flexibility, student 

achievement and improving efficiency of curriculum delivery processes. 

The case studies were also driven by the University of Bradford’s Academic Framework 

2009-14, which identifies diagnostic assessment, formative assessment and feedback as 

key activities within its approach to curriculum delivery. 

3. Generation of evidence: please describe how the reported research/evaluation findings were 

generated e.g. methods used 

This seminar first looked to the literature of educational development to consider what 

constitutes “assessment for learning”, and then focused on the literature of technologyenhanced 

learning to consider the pros and cons of carrying this out in an online 

environment. The work also drew on recent JISC-funded work in the sector, such as the 

FEASST and REAQ projects, which have sought to scope and explore best practice in e- 

assessment. 

The seminar presented research findings from two case studies carried out at the 

University of Bradford in 2008-10, to evaluate examples of e-assessment for learning, 

delivered in different ways by different lecturers, in search of a model of best practice for 

e-assessment for learning. The case studies come from different subject areas and adopted 

a mixed methods approach (Yin, 1984) consisting of structured interviews, pseudoqualitative 

surveys of student opinion through questionnaires, as well as analysis of 

quantitative assessment data; methodologically, the research is based on a pragmatic 

methodology to educational research (Pring, 2000). 

The first case study (Dr Liz Carpenter, Department of Clinical Sciences) investigated the 

quality of feedback-rich formative assessment on a biology foundation course, where 

students were presented with a formative e-assessment in the university’s e-assessment 

suite towards the end of their taught module, then had access to the same formative 

assessment and feedback via the virtual learning environment in the period leading up to 

their final e-assessment. Impact was measured in a number of ways: quantitative analysis 

of student progress; multiple questionnaires to evaluate student perceptions of the 

process and of their own study habits; analysis of student access patterns; comparison of 

student engagement with feedback and subsequent progress; comparison of e-feedback 

with face-to-face feedback. 



Page 2 of 9


The second case study (Dr Darwin Liang, Department of Engineering) demonstrated an 

innovative approach in teaching a first year engineering module. This “activity-based 

student-centred teaching and learning” (HEA Engineering subject centre Teaching Award 

finalist 2010) combines traditional face-to-face lectures and lab work with online quizzes 

and e- tutorials incorporating automated feedback, as well as regular low-stakes 

summative e-assessment and a final high-stakes summative e-assessment. For this case 

study, this approach was evaluated through statistical tracking of student achievement, 

structured interviews with students, questionnaires looking at student perceptions and 

attitudes, and also detailed in-depth reflections from the course tutor. 

The seminar concluded with a group discussion task where delegates from across the HE 

sector compiled lists of challenges facing formative e-assessment in the sector, and then 

prioritised these according to importance. These were subsequently written up and 

collated by the seminar facilitator, and fed back to the group after the event, and the 

results are included below. 

4. Existing evidence: please provide details of research/evaluation evidence drawn on and 

reported in the seminar 

There is a great deal of recent evidence from UK Higher Education, for example, from the 

National Union of Students (2009), the National Student Forum (2010) and the annual 

National Student Survey, indicating that feedback is a major issue in the sector. 

The literature from the field of educational development suggests that feedback can have a 

positive impact on learning, so long as the feedback is appropriate and timely, and is 

accompanied by opportunities for learners to reflect, act upon the feedback and build this 

into their future study (Black and William, 2009: Boud, 2000; Hattie and Timperley, 2007; 

Nicol and MacFarlane-Dick, 2006; Sadler, 1989). 

Literature on e-assessment (also referred to as “computer-assisted assessment” or 

“technology enhanced assessment”) suggests that automated formative feedback can be 

one of the key benefits of this aspect of technology enhanced learning (Bull and McKenna, 

2004; Gilbert et al, 2009; Nicol and Draper, 2009; Pachler et al, 2009; Whitelock and 

Brasher, 2006). There is consensus in the literature that this e-feedback can be efficient, 

and can save a great deal of instructor time, by delivering generic feedback to large 

numbers of students. Also, this feedback can offer a greater level of detail, and richer 

media, could even offer a level of flexibility and personalisation, and can be incorporated 

into a blended learning environment to support a diverse and changing student population. 

However, a number of key questions remain: is it possible to use this kind of automated 

generic-feedback to create “moments of contingency” (Black and William, 2009), especially 

for HE students? Can the use of automated, objectively marked tasks really provide 

appropriate feedback of sufficient quality to have a significant impact on learning for 

university students? Also, how will students perceive this kind of automated feedback; 

could it help to satisfy the demand from students for more (and better) feedback? 



Page 3 of 9


5. Research findings/new evidence: please describe any new findings or evidence reported in 

the seminar. 

Case study 1: Measuring the impact of formative e-assessment and feedback on learning. 

Dr Liz Carpenter, Department of Clinical Sciences, School of Life Sciences, University of 

Bradford. 

This case study drew a number of findings about the impact of formative e-assessment, 

which can be summarised as follows: 

Students do value feedback-rich formative e-assessments and especially appreciate 

having the feedback immediately. 

There is a significant association between student progress and engagement with 

formative tasks; that is to say, the students who engage with the e-feedback task 

make more progress than those who do not (although the number of times they do 

so does not make a significant difference). 

The quantity of engagement with the formative feedback is not significantly 

influenced by the student’s initial level, which suggests that this is not simply a case 

of the stronger students exhibiting higher levels of autonomy and learning skills. 

Student engagement is greatest during the period immediately prior to the 

summative examination. 

Access to formative tasks tends to be via laptops at home or in halls. 

Students who view the formative and feedback as part of their learning show the 

greatest amount of progress; students who see the formative tasks as mere 

preparation for the summative exam, or as an evaluation tool for their other 

revision, tend to benefit less. 

In conclusion, computers can deliver quality feedback. 

Case Study 2: Activity-based student-centred teaching and learning. Dr Darwin Liang, 

School of Engineering, Design and Technology, University of Bradford. 

The findings emerging from this case study can be summarised thus: 

Students enjoyed learning at their own pace and liked receiving automatic 

immediate feedback on their work. 

E-tutorials, with automated generic feedback, provide one more method to enable 

students to learn, with more opportunity for practice, greater flexibility and can be 

repeated on demand. 

Whilst there was no immediate statistically significant increase on student 

achievement, the impact of this teaching method is more likely to be seen in longer 

term student development. 

There were mixed reactions among students to the innovative teaching method, 

due to the diversity of students within the cohort. 

Students are sometimes more concerned about marks than about learning, and 



Page 4 of 9


need to have a tangible reward for their efforts. 

There is a risk that too many different e-assessment tasks during the semester may 

lead to “over-assessment” and a negative impact on student perceptions and 

learning. 

E-feedback tasks should not be too long or over complicated, because students tend 

to give up if the task is too time-consuming. 

Students are keen to be allowed to take responsibility for some of their learning and 

to support one another during learning. 

Students do appreciate the personal, face-to-face contact with the lecturer and do 

not want this to be replaced by automated tasks. 

6. Outcomes of research /evaluation evidence and the implications for policy and practice: 

please identify any application or outcomes of research/evaluation evidence and detail the 

implications for policy and practice for different stakeholder groups such as: academics, 

learning technology practitioners, professional developers, senior managers, policy makers, 

students, sector organisations, employers and professional bodies. 

The main conclusion of the research emerging from these case studies is that by following 

the Black and William (2009) definition of formative assessment, using automated 

feedback to create “moments of contingency”, there is certainly a suggestion that this can 

direct learners towards improved learning in the higher education sector. This is especially 

the case in foundation degree and first year undergraduate courses, where the course 

content may lend itself better to this kind of learning; also, incidentally, this is often where 

the largest student cohorts are to be found, so the efficiencies in terms of time saving will 

also be maximised. 

The main implication is that academics, learning technologists and senior managers should 

work together to place a greater emphasis on the potential of e-assessment for learning, 

not just e-assessment of learning. The most effective impact of e-assessment for the higher 

education sector may not in fact be for large-scale invigilated summative assessments in 

vast computer clusters, but rather for more flexible delivery of formative e-assessment 

tasks, where large numbers of students can receive immediate generic feedback. 

7. Emerging themes: please detail the discussion topics or themes that were raised by 

delegates during the course of the seminar - suggesting areas that would merit further 

investigation. 

The case studies prompted an interesting and wide-ranging discussion of themes related 

for e-assessment for learning. Much of the discussion focused on how to engage learners 

with the e-feedback: it is not sufficient to provide the feedback, but there must be a 

deliberate effort to develop a second reflective stage which transforms passive feedback 

into active “feedforward”. This concurs with the findings of the University of Strathclyde’s 

REAP project (Re-engineering Assessment practices in Higher Education) and the Australian 

Learning and Teaching Council’s Assessment 2020: Seven propositions for assessment 

reform in higher education , which both stress the importance of feedback being a 



Page 5 of 9


dialogue between the instructor and the student, and encourage peer feedback. It was 

suggested by delegates, for example, that students might be included in the question 

design process, creating their own questions and feedback to share with one another. 

There was also some discussion about the most effective way of generating a bank of 

formative questions, with associated feedback, as it was recognised that question creation 

can be a very time-consuming activity indeed. It was felt that it would be desirable to 

develop subject-specific banks of questions to share as an open educational resource 

across the sector. 

It was also stressed that regular feedback throughout the course is likely to have a greater 

impact than a one-off formative assessment, and there was also some discussion about the 

desirability of using low-stakes grades for these assessments as a way of engaging students 

with the formative tasks. These points were also further developed in the group discussion 

task which followed. 

After the case studies and the ensuing discussion, there was a group discussion task 

around the theme “What are the biggest challenges facing formative e-assessment in the 

HE sector”. Delegates broke into small groups of 3-5 participants to identify challenges and 

wrote them on post-it notes, then positioned these in order of importance on a flip-chart; 

these data were later collated for distribution to the group by email. 

The output from the discussion task may be summarised thus: 

One key challenge was “what constitutes effective feedback”? Does e-assessment 

really improve learning? Some wondered whether it was actually a good idea to 

provide a direct link to the correct answer, and others pointed out that there was a 

general lack of reading around the subject by students, which needed to be 

considered when writing feedback. 

Student engagement was identified as a key challenge, especially how to get 

students to engage with and reflect upon feedback, online and otherwise. It was 

also pointed out that we also need to have mechanisms to encourage discussion 

and create dialogue about the feedback. In addition, automated feedback offers 

limited support for social learning and we need to maintain the personalised aspect 

of feedback to be able to motivate learners in need of support. 

There was also discussion of mixed formative/summative assessments, where low 

stakes grades are assigned to encourage engagement with feedback, but may raise 

concerns about cheating. 

Staff and institutional engagement were also identified as key issues: staff can be 

reliant on technical specialists, there might be ineffective tools available, and the 

technology might be a hurdle. As well as having to learn the specific e-assessment 

software, staff also need to develop skills in question design and understanding the 

issues related to formative feedback. All of this can be very time consuming, 

especially at first. It would help if we could share question banks with other 

academics, but this can be a challenge, too. 

There was some concern that formative e-assessment may be more relevant in 



Page 6 of 9


some subject areas than others, and that it may be difficult in more discursive 

subject areas. We also need to consider how to deliver feedback for open-ended 

questions and answers which are not black & white, but require interpretation. It is 

not always appropriate for the teacher to make assumptions about why a certain 

answer is wrong. Also, work-based learners may have different needs. 

It was also pointed out that formative e-assessment should be viewed as part of a 

larger portfolio of methods of formative assessment, and we also need to consider 

the feedback we give to manually marked assignments, submitted electronically. 

In conclusion, the case studies and the ensuing discussion from this seminar raised a 

number of questions which merit further study and consideration: 

What exactly constitutes quality automated electronic feedback? 

Might topic-based feedback more useful to students than feedback on individual 

questions? 

Is formative e-assessment more applicable to some subject areas than others? Is it 

restricted to foundation/first year undergraduates? 

How do we best train students to engage with this feedback and make the most of 

it in their learning? 

How can academics work together to share their banks of questions and feedback? 

What are the best research methods to investigate these issues; how do we really know 

what students are doing with the feedback? 

8. Any other comments: please use this box to include any additional details. 

This half-day seminar in the HEA Evidence-based practice series 2010 took place at the 

University of Bradford on the afternoon of 8 th December 2010, in the Learn Higher room 

and the Richmond Building e-Assessment suite. 

The event was attended by 21 delegates representing 10 different UK HE institutions as 

well as the Higher Education Academy. Thanks are due to the delegates who, in many 

cases, braved inclement weather conditions to be able to attend. 

Special thanks and acknowledgements go to colleagues at the University of Bradford, 

without whose support this seminar would not have been possible: 

Professor Nigel Lindsey, Director of Learning and Teaching 

Dr Liz Carpenter, Department of Clinical Sciences 

Dr Darwin Liang, Department of Engineering 

Debbie Alstead and Professor Peter Hartley, Centre for Educational Development 

Also thanks are due to Clare Gash and Eddie Gulc from the Higher Education Academy for 

all their support, before, during and after the day of the event. 



Page 7 of 9


9. Bibliography/references (preferably annotated): please list any references mentioned 

in or associated with the seminar topic. Where possible, please annotate the list to enable 

readers to identify the most relevant materials. 

Black, P. and William D. (2009) “Developing the theory of formative assessment.” 

Assessment, Evaluation and Accountability, 21 (1), 5-31. 

Bloom, B.S. (1956) Taxonomy of educational objectives, Handbook I: the Cognitive Domain. 

New York: David McKay Co. Inc. 

Boud, D. (2000) “Sustainable assessment: rethinking assessment for the learning society.” 

Studies in Continuing Education, 22 (2), 151-167. 

Boud, D. and Associates. (2010) Assessment 2020: Seven propositions for assessment 

reform in higher education. Sydney: Australian Learning and Teaching Council. 

Bull, J. and McKenna, C. (2004) Blueprint for Computer-Assisted Assessment. London: 

RoutledgeFalmer. 

Gilbert, L., Gale, V., Wills, G. and Warburton, B. (2009) JISC Report on e-Assessment Quality 

in UK Higher Education. Southampton: University of Southampton. 

Hattie, J. and Timperley, H. (2007) “The Power of Feedback.” Review of Educational 

Research, 77 (1), 81-112. 

Higher Education Funding Council for England (2009) Enhancing learning and teaching 

through the use of technology: a revised approach to HEFCE’s strategy for e-learning. 

Available online http://www.hefce.ac.uk/pubs/hefce/2009/09_12/ 

National Student Forum (2010) NSF Annual Report 2010. Available online at 

http://www.bis.gov.uk/assets/biscore/corporate/docs/n/10-p83-national-student-forumannual-report-2010 

National Union of Students. (2009) Assessment Purposes and Practices. NUS briefing 

paper. 

Nicol, D. and Draper, S. (2009) “A blueprint for transformational organisational change in 

higher education: REAP as a case study.” In Mayes, T., Morrison, D., Meller, H., Bullen, P., 

and Oliver, M. (eds) Education through technology-enhanced learning. Higher Education 

Academy. 

Nicol, D.J. and Macfarlane-Dick, D. (2006). “Formative assessment and self-regulated 

learning: A model and seven principles of good feedback practice.” Studies in Higher 

Education, 31(2), 199-218. 



Page 8 of 9


Pachler, N., Mellar, H., Daly, C., Mor, Y., and William, D. (2009) Scoping a vision for 

formative e-assessment. A project report for the Joint Information Systems Committee. 

London: WLE Centre. 

Pring, R. (2000) Philosophy of educational research. London: continuum 

Sadler, R. (1989) “Formative assessment and the design of instructional systems”. 

Instructional Science, 18, 119-144. 

Whitelock, D. and Brasher, A. (2006) Roadmap for e-Assessment. Joint Information Systems 

Committee. 

Yin, R. (1984) Case study research. Beverley Hills: Sage Publications. 



Page 9 of 9

10. Briefing Paper Template - Higher Education Academy

Create successful ePaper yourself

Delete template?

Save as template?