AHRC Assessment Panellists Guidance

Assessment Panellists’ Guidance
Contents
1. What is an Introducer?
2. The role of Introducers prior to the panel meeting
3. The role of Introducers at the panel meeting
4. Proposal grading and ranking at the panel meeting
5. After the panel meeting
Appendix A Grading Scale used by panellists and resubmission policy
**IMPORTANT: if you identify a conflict of interest with any of the applications
you have been assigned to introduce please contact UK SBS as soon as possible
as we will need to reassign the application to another panellist to introduce**
1. What is an introducer?
Assessment panels discuss proposals, assign final grades and rank proposals in order of
priority of funding. The ranking of proposals is based on the assessments by the panel
members and the overall research quality of the proposal.
To assist in this process, three panel members will have an ‘Introducer’ role for each
proposal and these will be nominated prior to the panel meeting. Each proposal will have
one First Introducer, one Second Introducer and one Supporting Introducer (previously
known as Reader). Their role at the meeting is to lead the discussion on the proposal,
and they will have prepared notes and have assigned the proposal an initial grade prior
to the meeting.
2. The role of Introducers prior to the panel meeting
You should have received a batch of proposals along with any additional information
regarding the proposals to be considered.
1 Version 2.0
October 2013

Assigning of roles
Included with the batch of proposals will be a list indicating your assigned role for each of
the proposals: First Introducer, Second Introducer, and Supporting Introducer
(previously Reader).
As you will be aware, it is not possible to achieve total coverage, within a single panel, of
the full range of subjects and the wide diversity of applications submitted to the AHRC.
Therefore, you may have been assigned some proposals that do not lie within your
precise area of subject expertise, but where you will still need to make an assessment of
the proposal against the aims of the scheme, guided by your experience as both a
reviewer and as a researcher.
For the proposals that you have been assigned the role of ‘Introducer’, you are asked to
read these proposals. Considering the assessment criteria provided in the scheme
guidance, and make your own assessment of the proposal assigning a grade and
producing notes to help your introduction and discussion of the proposal at the panel
meeting. The academic grade descriptors are detailed in Appendix A, but panellists
should always refer to the scheme guidance document as some schemes use different
grade descriptors.
Conflicts of interest
The AHRC will not assign you to an Introducer role where you are at the same institution
as the Principal Investigator (or Co-Investigator(s)). However, it is possible that you
have a connection with the PI or Co-I that we are not aware of and we would therefore
ask that, on receiving the set of proposals, you have a quick look at those to which you
have been assigned a role and check whether such a conflict exists.
If this is the case please alert us as soon as possible, in order that we can reassign the
proposal to another panel member in time for them to read and consider the proposal
prior to the meeting.
Academic assessments
When making an assessment of the proposals, you are carrying out the same kind of
assessment, with the same rigour, as if you were acting as a peer reviewer, except that
2 Version 2.0
October 2013

you are addressing your comments to the panel rather than to the applicants, as there
will not be a PI response stage.
In order to fully understand the quality and content of the proposals, you should
familiarise yourself with the aims and objectives and the assessment criteria for the
scheme. A copy of the scheme guidance should be included with the panel papers. If this
is not included, or if you have any questions about the scheme which you are assessing,
please contact the AHRC officer responsible for the panel meeting.
You should:
• ensure you have read the entire proposal thoroughly
• familiarise yourself with the strategic aims of the AHRC and the aims of the
scheme for the proposal that you are assessing
• be aware of the full range of grades and their descriptors at your disposal
• contact staff at the AHRC if anything is unclear.
Analysis
You should:
• always provide evidence to support your observations. Use only the information
provided in the application form
• take into account the information you are asked to provide under each heading or
item in the scheme assessment criteria. Ensure sufficient detail is provided for
each one
• give a clear assessment of strengths and weaknesses of the proposal and indicate
whether these are major or minor concerns
• provide an evaluation of the risks associated with the project
• contextualise the proposal that you are assessing within current work in the field,
and comment on its relative importance/significance
• be receptive to new ideas and approaches to thinking within your discipline as well
as methodology
3 Version 2.0
October 2013

• identify any inconsistencies and contradictions in the proposal
• in the case of interdisciplinary applications: do the different disciplines meet up in
a coherent way?
• provide enough information to enable a judgement on the relative quality of this
proposal compared to other applications.
General points
You should:
• provide an impartial, objective, fair and analytical assessment of the proposal
which you are reviewing
• ensure you are providing an evaluation, not a description of the work proposed
• ensure your grade is justified by, and consistent with, your comments.
All costs that are considered justified as reasonable for the research proposed are
allowable and should be accepted. Comments on the justification can only be considered
for:
• ‘Directly incurred’ costs
• The level of investigators’ effort (i.e. the time they are spending on the project)
• ‘Other Directly Allocated’ costs (except charge out costs for departmental
technicians and administrative services)
• ‘Exceptions’ costs (this is where costs relating to project studentships will appear).
Introducer Report Form or Scheme Assessment Form
You should complete an Introducer’s Report Form or Scheme Assessment Form for
each proposal where you have been assigned the role of Introducer. The form will act as
an aide memoire at the panel meeting – it does not need to be submitted prior to
the meeting. You should assign the proposal a grade 6 - 1 (see Appendix A for grade
descriptors) and an indicative ranking (see Ranking section below).
Ranking
Regardless of whether you are First or Second Introducer, Supporting Introducer or not
assigned to a proposal which you have read, it is acceptable to begin to think about a
4 Version 2.0
October 2013

elative rank for proposals which you have graded similarly. This will help you gauge
how you think applications fare against each other when ranking during the panel
meeting.
All proposals need to be graded but only those graded 4 or above need to be ranked.
Previous panels have found that using a decimal grading system for proposals is an
effective and efficient mechanism for the ranking of proposals. Therefore, Introducers
should award a decimal grade to each proposal with a view to the panel agreeing a final
decimal grade. This allows the Introducers to indicate the strength of their grading. For
example, a good 4 might become 4.6, a weak 5, 5.1 and an excellent 6 a 6.8 (this is
essentially a more nuanced version of High/Medium/Low). As you go through the
meeting it is permissible to revisit the earlier decimal places in light of other applications
assessed. The key thing is to make sure that the grade and its descriptor fit the quality of
the proposal and that the panel is content with the ranked order of the proposals. The
decimal grading is only a mechanism to aid the ranking of proposals and only the final
agreed grade will be fed back to the applicant. This does not in any way alter or replace
the current grading scales.
Paperwork for the meeting
Please remember to bring the following paperwork to the panel meeting:
• All the applications
• Introducer forms or Scheme Assessment Forms for all proposals where you are
First, Second or Supporting Introducer
3. The role of introducers at the panel meeting
Introducers will lead the discussion on a proposal. If you are the First Introducer, you
should give a brief one to two minute summary of the proposal to aid the panel
discussion. If you are the Second or Supporting Introducer then you should avoid
repetition, and confine your discussion of the proposal to points not already covered, or
where you disagree with the First or Second Introducers.
Introducers should present their assessments and grades of the proposals using the
Introducer Report or Scheme Assessment Form as an aide memoire for each proposal.
The AHRC will not collect these forms at the end of the meeting, unless the panel agrees
5 Version 2.0
October 2013

to give feedback to the applicant, in which case we may take Introducer Forms back to
the office to aid in drafting the text.
The grades assigned by the Introducers help the panel to reach a final grading decision
and to rank proposals in priority order for funding. They have no other value outside the
panel meeting and are not communicated to the applicant.
4. Proposal grading and ranking at the panel meeting
The Panel as a whole will agree an overall grade from 6-1 (please refer to Appendix A
for grade descriptors) for each proposal reached through discussion of the proposals –
taking into account the Introducers’ initial grades and comments, alongside the
comments of the panel as a whole. The overall grade will be used in determining the
proposal's relative ranking.
The panel should rank proposals relative to one another as they proceed through the
meeting. Some panels may find it useful to use decimals or high/mid/low when allocating
grades if they have a high number of proposals to rank e.g. 5.5 or 5 mid. At the end of
the meeting, the panel will review the ranking list before finally agreeing the ranked
order. Proposals given a 3, 2 or 1 grade do not need to be ranked.
The AHRC is committed to equal opportunities. Please ensure therefore that all proposals
are graded and ranked on equal terms.
Proposals are submitted to the AHRC in confidence and may contain confidential
information and personal data belonging to the applicant (and other researchers named
in the proposal). Please ensure therefore that all proposals are treated confidentially,
referring to the Peer Review College Handbook for further guidance on Confidentiality,
Data Protection, and Freedom of Information:
http://www.ahrc.ac.uk/Peer-Review-College/PRC-Handbook/Pages/PRC-Handbook.aspx
5. After the panel meeting
After the panel meetings, the final funding decisions will be made by a Senior AHRC
Officer on the basis of the ranked list provided and the level of funding available. It is,
therefore, vital that panel members do not divulge or discuss panel meeting outcomes
with individuals outside the meeting. Maintaining confidentiality is of paramount
importance.
6 Version 2.0
October 2013

All announcements of outcomes and funding decisions will be made by the AHRC. Any
panel member who is asked directly for feedback by applicants should refuse and advise
applicants to direct all such requests to the AHRC.
7 Version 2.0
October 2013

Appendix A
The Grading Scale for Academic Reviewers and Panellists is contained in the table below:
Grade
Descriptor – Research Grants Scheme
An outstanding proposal that is world-leading in all of the following:
scholarship, originality, quality, and significance. It fully meets all the
6
assessment criteria for the scheme and excels in many or all of these. It
provides full and consistent evidence and justification for the proposal and
management arrangements are clear and convincing.
It should be funded as a matter of the very highest priority.
A proposal that is internationally excellent in all of the following: scholarship,
originality quality and significance. It fully meets or surpasses all the
assessment criteria for the scheme. It provides full and consistent evidence
5
and justification for the proposal and management arrangements are clear
and convincing.
It should be funded as a matter of priority, but does not merit the very
highest priority rating.
A very good proposal demonstrating high international standards of
scholarship, originality, quality and significance. It meets all the assessment
4
criteria for the scheme. It provides good evidence and justification for the
proposal and management arrangements are clear and sound.
It is worthy of consideration for funding.
A satisfactory proposal in terms of the overall standard of scholarship and
quality but which is not internationally competitive and/or which is more
limited in terms of originality/innovation, significance and/or its contribution
to the research field. It satisfies at least minimum requirements in relation to
3
the assessment criteria for the scheme, provides reasonable evidence and
justification for the proposal and management arrangements are adequate
overall.
In a competitive context, the proposal is not considered of sufficient priority
to recommend for funding.
A proposal of inconsistent quality which has some strengths, innovative ideas
and/or good components or dimensions, but also has significant weaknesses
or flaws in one or more of the following: conceptualisation, design,
2
methodology, and/or management. As a result of the flaws or weaknesses
identified the proposal is not considered to be of fundable quality. A proposal
would also be graded 2 if it does not meet all the assessment criteria for the
scheme.
8 Version 2.0
October 2013

It is not recommended for funding.
A proposal which falls into one or more of the following categories:
• has unsatisfactory levels of originality, quality and/or significance
• falls significantly short of meeting the assessment criteria for the scheme
1
• contains insufficient evidence and justification for the proposal
• displays limited potential to advance the research field
• has potential outcomes or outputs do not merit the levels of funding sought
Grade
6
5
4
3
• is unconvincing in terms of its management arrangements or capacity to
deliver the proposed activities.
It is not suitable for funding.
Descriptor – Research Grants Scheme
An outstanding proposal that is world-leading in all of the following:
scholarship, originality, quality, and significance. It fully meets all the
assessment criteria for the scheme and excels in many or all of these. It
provides full and consistent evidence and justification for the proposal and
management arrangements are clear and convincing.
It should be funded as a matter of the very highest priority.
A proposal that is internationally excellent in all of the following: scholarship,
originality quality and significance. It fully meets or surpasses all the
assessment criteria for the scheme. It provides full and consistent evidence
and justification for the proposal and management arrangements are clear
and convincing.
It should be funded as a matter of priority, but does not merit the very
highest priority rating.
A very good proposal demonstrating high international standards of
scholarship, originality, quality and significance. It meets all the assessment
criteria for the scheme. It provides good evidence and justification for the
proposal and management arrangements are clear and sound.
It is worthy of consideration for funding.
A satisfactory proposal in terms of the overall standard of scholarship and
quality but which is not internationally competitive and/or which is more
limited in terms of originality/innovation, significance and/or its contribution
9 Version 2.0
October 2013

2
to the research field. It satisfies at least minimum requirements in relation to
the assessment criteria for the scheme, provides reasonable evidence and
justification for the proposal and management arrangements are adequate
overall.
In a competitive context, the proposal is not considered of sufficient priority
to recommend for funding.
A proposal of inconsistent quality which has some strengths, innovative ideas
and/or good components or dimensions, but also has significant weaknesses
or flaws in one or more of the following: conceptualisation, design,
methodology, and/or management. As a result of the flaws or weaknesses
identified the proposal is not considered to be of fundable quality. A proposal
would also be graded 2 if it does not meet all the assessment criteria for the
scheme.
It is not recommended for funding.
A proposal which falls into one or more of the following categories:
• has unsatisfactory levels of originality, quality and/or significance
• falls significantly short of meeting the assessment criteria for the scheme
1
• contains insufficient evidence and justification for the proposal
• displays limited potential to advance the research field
• has potential outcomes or outputs do not merit the levels of funding sought
• is unconvincing in terms of its management arrangements or capacity to
deliver the proposed activities.
It is not suitable for funding.
Resubmission policy
In line with the AHRC’s approach to demand management, unsuccessful applicants will
no longer be permitted to resubmit the same, or substantively similar, proposal to the
same scheme. In very particular circumstances the AHRC may, exceptionally, decide to
invite the applicant to resubmit the proposal. This will happen only where the panel
identifies an application of exceptional potential and can identify specific changes to the
10 Version 2.0
October 2013

application that could significantly enhance its competitiveness. Invited resubmissions
will be assessed in the usual way in competition with all other proposals.
In addition, there are schemes/calls where invited resubmission is not an option due to
the particular nature of the scheme/call, i.e. because the call is a one off and there will
be no further rounds under which to submit applications. In such cases the panel may
wish to consider the possibility of a conditional award, where appropriate.
11 Version 2.0
October 2013