Pacific Research Expert Advisory Group Criteria for the

tec.govt.nz

Pacific Research Expert Advisory Group Criteria for the

Performance-Based Research Fund

Pacific Research Expert Advisory Group

Criteria for the 2012 Quality Evaluation

Pacific Research Expert Advisory Group Criteria – PBRF 2012 Quality Evaluation Page 1 of 9


Background

After the Performance-Based Research Fund (PBRF) 2006 Quality Evaluation, a

Sector Reference Group (SRG) was established to consider feedback from the

sector and lead the redesign of the PBRF before the 2012 Quality Evaluation.

A major recommendation from the SRG was to establish a Pacific Research Expert

Advisory Group (EAG) at the same time as establishing the peer review panels. It

was also recommended that the criteria forPacific Research’ used in the PBRF 2003

and 2006 Quality Evaluations be revised.

Purpose

The peer review panels will assess and score Evidence Portfolios (EPs) submitted to

the PBRF 2012 Quality Evaluation. Where an EP has at least one Nominated

Research Output (NRO) that meets the criteria set out by the Pacific Research EAG

additional advice can be sought by referral to the Pacific Research EAG and a score

and an opinion on the EP will be provided back to the primary peer review panel

making an assessment of the EP.

The Pacific Research EAG has developed criteria to assist TEOs and researchers

with determining whether or not their EPs demonstrate the characteristics of Pacific

Research that are appropriate for referral to the Pacific Research EAG.

Pacific research encompasses research that involves specific ethnic groups within

the Pacific as well as Pacific research that spans Pacific communities. The term

Pacific research” as used in this document relates to all Pacific research, unless

indicated otherwise. Pacific research may be undertaken and presented by Pacific or

non-Pacific researchers.

The Pacific Research EAG criteria must be read in conjunction with the PBRF Quality

Evaluation Guidelines 2012 (general Guidelines) and the panel-specific guidelines of

the relevant panels. The Pacific Research EAG criteria do not replace or supersede

the requirements for EPs that are set out in the general Guidelines: they are

complementary to the panel-specific guidelines.

The indicative timetable for publication of the Pacific Research EAG criteria is

outlined below:

Draft criteria developed August 2011

Publication on TEC website for sector

consultation

September 2011

Feedback and revision of criteria Mid September 2011

Final criteria published End September 2011

Pacific Research Expert Advisory Group Criteria – PBRF 2012 Quality Evaluation Page 2 of 9


Pacific Research EAG criteria

General Information

The EAG will assess the significance, quality and impact of research which is

identified as Pacific research and will provide a score and an opinion to the peer

review panels accordingly.

The Pacific Research EAG has provided elaborations on the general Guidelines. The

following perspectives have informed these elaborations:

• The impact of Pacific research on Pacific communities, its relevance to, and

uptake by, those communities is particularly important as this reflects a

commitment by Pacific researchers to research which will benefit communities.

For this reason, Pacific research may be more likely to be of an applied nature

than some other kinds of research.

• Contemporary Pacific research and discourse on Pacific research are

emerging. As a result, there are a small number of Pacific researchers and

those with significant research experience often commit significant resources to

developing new and emerging Pacific researchers.

Pacific research is reflective of the traditions of the past, as well as the present

and future. It often embodies paradigms, perspectives and critical stances that

are not always captured in mainstream research.

Pacific research is an inclusive concept, incorporating research approaches

that are both ethnic-specific and Pan Pacific in scope.

Criteria for EPs to be referred to the Pacific Research EAG

In order to meet a test of substantiveness for referral to the Pacific Research EAG at

least one NRO must be designated in the Evidence Portfolio (EP) 1 as being of a

Pacific research nature. The PBRF gives due emphasis to both research by Pacific

researchers and research into Pacific matters. Such research may also

acknowledge different approaches to the research process.

The Pacific Research EAG will receive EPs for assessment via two possible

pathways:

• TEOs/ researchers may request assessment of their EPs to the Pacific Research

EAG.

• The Chair of a peer review panel may refer an EP to the Pacific Research EAG

for consideration.

1 A field will be available in the PBRF IT system to identify the relevant NRO(s). When submitting EPs,

TEOs must clearly identify which NRO meets the criteria for consideration by the EAG.

Pacific Research Expert Advisory Group Criteria – PBRF 2012 Quality Evaluation Page 3 of 9


As required by the general Guidelines, all EPs referred to the EAG will be considered

by the EAG. If, however, the EAG is unable to return an opinion on the EP, the

Pacific Research EAG Chair will advise the Chair of the peer review assessment

panel.

Expected types of research

Pacific research may be embodied in any of the types of research output listed in the

general Guidelines. Multiple ways of presenting and disseminating research for

different audiences are common (for example, one major research project may result

in a number of different outputs and be presented through a range of channels). Full

and equivalent consideration will be given to all types of research output.

The general Guidelines describe the types of research output that are permissible

(see Chapter 2, Section C: Types of Research Output).

Pacific research is a newly documented field, relative to many mainstream research

fields, and research outputs are commonly presented as: oral presentations and

addresses, working papers and web-based presentations. It is understood that

research processes that require validation by the community may be lengthier than

normally expected. This may affect the quantity and types of research output

produced.

In order for a TEO to request referral of an EP to the Pacific Research EAG, the EP

must contain at least one NRO that meets the definition of Pacific research.

The general Guidelines describe the types of research output that are permissible

(see Chapter 2, Section C: Types of Research Output).

In addition to standard journal articles and conference papers, books etc., Pacific

outputs that are generated out of research will often include:

• Occasional papers, working papers

• Oral presentation or address

• Composition

• Performance (including choreography)

• Traditional dance, theatre, story-telling

• Literature (novels, poetry, etc)

• Art work

• Reports and presentations to the community

The presentation of research through oral forms (such as an address) is often very

important, since the person and the delivery are considered a crucial part of the

research engagement with the community. Oral outputs must be electronically

available for review, if required; Quality of an oral presentation may be assured by a

Pacific Research Expert Advisory Group Criteria – PBRF 2012 Quality Evaluation Page 4 of 9


senior individual witnessing the presentation (for example by a scholar of renowned

repute, or an expert with appropriate academic credentials)

Applied research and action research are common approaches in Pacific research.

These approaches may result in new service models, in which case the associated

documentation may be submitted as a research output, provided that it falls within

the definition from the General Guidelines.

Research-based dictionaries and translations are valid research outputs. On the

other hand, other forms of teaching material are not valid research outputs, unless

these were produced as a result of research that conforms to the definition in the

general Guidelines

Elaboration of the definition of research

As with all research qualifying for assessment in the PBRF, Pacific research must

fulfil the PBRF definition of research in the general Guidelines (see Chapter 1,

Section D: What Counts as Research).

Within this definition, however, Pacific research is seen as a broad descriptor that

covers a wide range of subject areas and includes various Pacific approaches to

research. It is expected that much of the research will be multi-disciplinary, and may

include a range of methodological approaches.

Pacific research is broadly characterised as follows: an EP or NRO that includes

Pacific research will demonstrate some, or all, of the following characteristics, and

should show a clear relationship with Pacific values, knowledge bases and a Pacific

group or community.

Paradigm

Pacific research may be characterised by one or more of the following:

• It Is informed by and embedded within the continuum of Pacific world views,

knowledge, practices and values

• It is conducted in accordance with Pacific ethical standards, values and

aspirations, such as responsiveness and reciprocity

• It includes research carried out and/or reported in Pasifika languages such as

Niuean, Samoan for example

• It comprises a range of disciplines and disciplinary research practices including

archival and historical research, sciences and social sciences

• It involves research processes and practices that are consistent with Pacific

values, standards and expectations

• It includes methods, analysis and measurements that recognise Pacific

philosophy, spirituality and experience.

Pacific Research Expert Advisory Group Criteria – PBRF 2012 Quality Evaluation Page 5 of 9


Participation

Pacific research is likely to:

• Involve the active participation of Pacific people (as researchers, advisors and

/or stakeholders)

• Demonstrate community engagement - it recognizes and validates the

relationship between the researcher and the ‘researched’

• Engage the Pacific community right from the initial stages of the research

• Develop post-graduate capacity including the mentoring of Pacific postgraduates

and emerging researchers

• Involve the organisation of conferences/workshops which focus on Pacific

research and researchers.

Contribution

Pacific research is likely to:

• Be relevant and responsive to the needs of Pacific peoples

• Have a demonstrable impact on Pacific peoples

• Contribute to and enhance the Pacific knowledge base in the relevant subject

area

• Contribute to a greater understanding of Pacific cultures’ experiences and world

views

• Protect Pacific knowledge

• Contribute to Pacific knowledge, spirituality, development and advancement

• Be responsive to changing Pacific contexts.

Capacity and capability

Pacific research:

• Builds the capacity and capability of Pacific researchers

• Builds non-Pacific researchers’ knowledge and understanding of Pacific

research paradigms and issues

• Enhances the capacity of Pacific communities to access and use the results of

the research.

Research that falls within the broad ambit of Pacific research, as outlined above, may

be undertaken by Pacific and non-Pacific people.

General expectations for standard of evidence to be supplied

Pacific research covers a wide range of subject areas and can result in many types

of research output. In cases where the nature of the quality assurance process or the

Pacific Research Expert Advisory Group Criteria – PBRF 2012 Quality Evaluation Page 6 of 9


channel for dissemination of a NRO may not be standard within the discipline, staff

members are advised to provide information as to the quality assurance processes

used, and the dissemination channel associated with the research output. This

information should be provided in the ‘Other Comments’ field of the EP.

The Pacific Research EAG will not seek additional verification of information.

While conventional methods of quality assurance (such as peer-review of journals,

and curation of exhibitions) will apply to Pacific research, other quality assurance

processes may also apply. One measure of quality assurance for Pacific research

may be the extent to which this has been disseminated to the community, and

evidence of feedback has been gathered from the community, prior to wider

dissemination. Dissemination to Pacific communities is a highly respected and well

regarded mechanism for communicating the results of research. For example, Pacific

health research may have important implications for communities, and to achieve a

high level of awareness, critique and uptake of this type of research may require

appropriate dissemination to the communities in question.

Sometimes there is a delay in receiving feedback, and acknowledgement of the

research occurs sometime in the future. The effort required in targeting and

disseminating Pacific research, and the quality of dissemination channels

themselves, may vary. Researchers should, therefore, describe the type of approach

used to disseminate research, including targeted dissemination, and where possible,

describe any evidence of feedback or acknowledgement that may indicate quality. It

is important that the EAG receives advice as to the relative quality of the

dissemination channels and the rigour of quality assurance processes presented in

an EP for any Pacific research.

Indicators of research quality for Pacific research may include:

• endorsement by community leadership, prior to wider dissemination

• endorsement through fono, or Pacific media (recognising that these may be

community, national, regional or Pan Pacific), prior to wider dissemination

• evidence of dissemination and uptake of research findings by Pacific regional

media, and Pacific research communities

• endorsement and uptake across Pacific communities.

In some cases, information that demonstrates the significance and impact of the staff

member’s research are examples of peer esteem, and as such, may be included in

the Peer Esteem Component of the EP.

Peer review panels should note that the opportunities for publication of Pacific

research in mainstream journals, or through other mainstream dissemination

channels, may be limited. Pacific journals, which may not be widely known, are

nevertheless important sites for publishing because they reach Pacific communities,

including communities of Pacific academics. Increasingly, such channels are

developing their own quality assurance processes. In addition, Pacific research is

Pacific Research Expert Advisory Group Criteria – PBRF 2012 Quality Evaluation Page 7 of 9


often published as occasional papers, on websites and through a variety of Pacific

media. For the evaluation of a NRO in a Pacific language:

• if the NRO is focused on the analysis of Pacific cultures, concepts, values, or

methodologies, it should be evaluated in its original language by the relevant

subject panel (with the advice from a Pacific Research EAG member with skills

in the language and subject area, as required).

• if the subject matter of the NRO will not be compromised in meaning, it may be

provided to the relevant subject panel, if requested, in an English translation.

Use of additional expert advice

The Pacific Research EAG Chair may draw on additional advice from outside the

immediate EAG membership in order to address specialist content of any given EP.

This may include the appointment of Specialist Advisers, as required. For example it

may be necessary to seek additional advice when group members of the EAG have a

conflict of interest with a particular EP.

Individual staff members submitting EPs may not request that the EAG seek

additional advice.

The EAG may access specialist advice to assist in assessing NROs wholly or

partially in a language that is inaccessible to EAG members.

Assessment of an EP by the Pacific Research EAG

The Pacific Research EAG Chair will assign an EP to one or two EAG members for

assessment

A 0-7 points scoring system will be used to provide one score for an overall

assessment of each EP that meets the criteria for referral to the Pacific Research

EAG. The scoring and criteria for assessment is detailed in the following section.

The peer review panel will consider the score and opinion provided by the Pacific

Research EAG.

The peer review panel will determine the final Quality Category assigned to that EP,

taking into account the score and opinion of the Pacific Research EAG. Accordingly,

the score and opinion of the Pacific Research EAG may reduce or improve or have

no effect on the scores assigned by the panellists.

See the general Guidelines, Chapter 3, Section C: Assessing and Scoring the Three

Components of an EP.

The Moderators will be responsible for ensuring that there is consistency in the way

the peer review panels consider the advice of the Pacific Research EAG.

Pacific Research Expert Advisory Group Criteria – PBRF 2012 Quality Evaluation Page 8 of 9


Pacific Research EAG – criteria and scoring guide

Each of these criteria will be given an appropriate weight on a case by case basis; not all will apply in every discipline and only the most

relevant will be considered.

Criterion 1 3 5 7

Research question/theory/

methodology directly relevant

to policy/practice/discipline in

substantive Pacific research area

Not relevant

Some relevance in both

question methodology/theory

Significant level of relevance

in both question/ methodology/

theory

Completely relevant and

directly able to affect policy/

practice in methodology/

theory

Active dissemination of research

findings in ways likely to affect

discipline, policy/practice, by Govt,

Pacific community, practitioners and

research

No relevant

dissemination

activity (including

non-refereed

publications)

Limited level of dissemination

activity (including local

refereed journals and

publications)

Significant level of relevant

dissemination activity

Wide-scale dissemination of

findings in ways likely to

affect policy/practice

New insights and understanding

into substantive Pacific issues by

Govt, Pacific community,

practitioners and researchers

No new insights

Minor changes in

understanding of issue

Significant changes in

understanding of issue

Paradigm shift in

understanding of issue

Change in policy/practice/discipline

relevant to substantive Pacific issue

No change Minor changes Significant changes Major shift in policy/practice

Pacific Research Expert Advisory Group Criteria – PBRF 2012 Quality Evaluation Page 9 of 9

More magazines by this user
Similar magazines