European Peer Review Guide - European Science Foundation

36
European Peer Review Guide
4.10.3 Measure of quality and usability
of the reviewers’ assessments
Some of the items defining the quality and usability
of the assessments made by individual/remote
reviewers are:
• Conflicts of interest;
• Completeness;
• Comprehensibility and clarity;
• Appropriateness of the language used;
• Fit for purpose;
• Timely;
• Substantiated judgments and scores.
4.10.4 Evaluation
Evaluation entails appropriate measures and means
of supervising and scrutinising the process and its
implementation by authoritative and experienced
individuals or groups of individuals. This could
comprise parties either internal or external to the
organisation or a mixture of the two. The term ‘evaluation’
used here does not refer to ex-post evaluation
of the funded research38.
It is important to clearly describe to all relevant
parties and at the beginning of the process the following
items:
• The purpose of the evaluation;
• The scope of the evaluation;
• The means available to conduct the evaluation;
• What could be the outcome of the evaluation.
4.10.5 Overall recommended measures
in support of quality assurance
To support quality assurance the following aspects
may be considered:
• Identify and mandate dedicated individuals or
groups of individuals responsible for the conceptualisation
and administration of quality reviews; as
far as possible, ensure continuity by avoiding the
use of temporary assignments and frequent staff
changes. Make clear the roles and the responsibilities
of the programme officers and administrators
and thus demand accountability;
• Ensure consistency and clarity of the published
material and all other communication streams to
all stakeholders;
• Offer clear instructions, briefing notes and, if pos-
38. Ex-post evaluation of the funded research has not been included
as part of this Guide. On this topic see, for example, the Reports
of the ESF Member Organisation Forum on Evaluation of Publicly
Funded Research at: http://www.esf.org/activities/mo-fora/
evaluation-of-publicly-funded-research.html and the ESF Member
Organisation Forum on Evaluation of Funding Schemes and
Research Programmes at: http://www.esf.org/activities/mo-fora/
completed-mo-fora/evaluation-of-funding-schemes-and-researchprogrammes.html
sible, training sessions for reviewers and panel
members to ensure the coherence and consistency
of their approaches;
• Keep the procedure as simple as possible, increase
the level of standardisation and automation
whenever proven technologies and resources are
available. Systematic tracking of reviewers’ quality
can be very beneficial.
• Conduct periodic reviews of the processes and procedures.
The cycle length of the reviews – whether
they are programme-based, department/unitbased
or institution-based – may vary according
to disciplinary or institutional needs.
The survey on peer review practices has shown that
the responding organisations adopt the following
correcting actions in cases when the quality and
usability of the assessments fall short of their standards:
• The entire review may be discarded and not used
according to 56% of the respondents;
• The review might be returned to the reviewer for
completion/additional information (according
to 52% of the respondents) or for modification
(according to 32%);
• 40% of the responding organisations indicated
that reviewers may be tagged based on the quality
and usability of their assessments39 with qualifying
information that may be used for future
references;
• The data protection laws of each country may dictate
the nature and usage of this information.
4.11 Variants of funding
instruments and their implication
for Peer Review
One of the main challenges for structuring both the
Guide and the supporting peer review survey has
been to categorise main funding instruments common
to European research funding and performing
organisations and councils. The conclusion has been
to treat the task of grouping of instruments along
two dimensions.
The first dimension considers the main typology
of the funding instruments that is driven only by
the nature and size of the funding opportunity; the
second dimension relates to the different program-
39. See European Science Foundation (2010b), ESF Survey
Analysis Report on Peer Review Practices, in particular Section 3.2,
§3.2.2, Question 22: “What concrete actions can result from the
evaluation of a review’s quality and usability by your organisation?”
(Table 3.7).