EIPOT Final Project Report - Stockholm Environment Institute

ERA-NET SKEP Project EIPOT (www.eipot.eu)
“Development of a methodology for the assessment of global environmental impacts of traded goods and services”
Robust
• Defensible theory
• Sensitivity
• Data quality
• Reliability
• Consistency
• Comparability
• Boundaries
The EIPOT document on the RACER evaluation framework (Lutter and Giljum 2008) provides a more
detailed description and explanation of these sub-criteria.
3.2 RACER evaluation framework
Our proposed evaluation framework enables an analysis of methods qualitatively and quantitatively.
The qualitative step provides, for each sub-criterion, a description of advantages and disadvantages of
the methods. In the second step, these descriptions are quantified by allocating numerical scores to
each of the sub-criteria, on a three-level scale between zero and two. This scoring can be used to rate
whether a method does not fulfil a criterion at all (zero), fulfils it only partly (one), or is appropriate to
answer the criterion’s question (two).
For each of the five main RACER criteria, an average score was calculated for each method. We
refrained from weighting these values or from summing them into one overall score. Instead, we
compared the five individual RACER scores for each method to produce a more comprehensive
picture of the differences between methods. Presenting average scores in radar diagrams provides an
easily readable overview of the evaluation. An overall, weighted or unweighted, RACER score for
each of the methods was not calculated as the aggregation of major categories was considered too
ambiguous. Not all criteria are of equal importance for decision-making. If decision-making is to be
based on correct information, credibility and robustness may be better indicators than acceptance.
The fact that a method is accepted does not necessarily mean it is the best way to support decisions.
An unsuitable method could become accepted because the proponents are better marketers, have
more resources for dissemination or are better connected. Conversely, if a method is chosen that is
credible and robust, it can (on its merit) become accepted through good published information. Just
because a method is difficult does not mean it should receive lower priority than an easy method. If
the best method for decision-making is difficult, then it just means it has to be supported by experts.
One example is perhaps the support for economic policy decisions that are often based on information
provided by a handful of economists who use general equilibrium models or econometric models.
These are difficult to implement and use, but they are the state-of-the-art in economic modelling. It
would not be advisable to replace these with a more simple method just because the latter was easier.
These considerations would make a reasonable argument for multiplying the individual RACER scores
with different weights, arguably higher for ‘relevance’, ‘credibility’ and ‘robustness’ and lower for ‘easy’
and ‘acceptance’. However, there is no widely accepted consensus about the value of these weights
and it was therefore decided not to apply weighting factors. Instead, qualitative considerations were
used in specifying a method, enabling us to distinguish between the performance of different
methodologies with respect to categories regarded as of more or less importance to the project’s aim.
This means that the methodology which had been ranked best in the RACER analysis is not
15