Successful transport decision-making - Osmose

FS 36: Measuring outcome indicators
Vol 2 - Table of Contents
Next
T11
What are outcome indicators?
Outcome indicators measure the impacts, benefits and
changes that are experienced by different stakeholder groups
during or after implementation of the project.
To determine what impact a project has had, it is necessary to
compare conditions 'after' implementation with those that
existed 'before' implementation, taking into account other
external factors that may have had an influence on the
observed outcome (e.g. changes in car ownership, or a
national publicity campaign). Thus, monitoring outcome
indicators needs to start before the project begins.
Outcome indicators measure what impact the project has had,
in terms of meeting underlying policy objectives. For example,
are more people cycling now? Has car use decreased? Has air
quality and traffic safety improved?
The role of outcome indicators in
evaluation
Post-implementation evaluation examines the consequences of
implementing a strategy or scheme, and how these relate to the
intended consequences that provided the justification for going
ahead with the project. Outcome indicators thus provide crucial
information about the performance of the project and, in
conjunction with data on resource consumption (i.e. input
indicators), enable factors such as cost effectiveness to be
assessed. Outcome indicators may need to be provided in the
short, intermediate and long-term. The timing of a postimplementation
evaluation (and the collection of the associated
outcome indicators) is important, because if it is carried out too
soon, the full impacts arising from the project may not be
captured (e.g. build up of patronage on a new tram line).
Conversely, if it is undertaken too late, resources will be wasted
and similar projects will not benefit from the lessons learnt.
Using outcome indicators
Before selecting indicators, establish:
Whether you need to compare your results with those from other
projects, or to conform to certain reporting formats (e.g. national
requirements for large infrastructure schemes);
Over what time period the outcome indicators need to be
measured;
How the impacts of the project will be determined (e.g. by
collecting 'before' and 'after' data; by collecting data in another
area to control for the influence of external factors, by
interviewing key stakeholders to assess their views on the
projects impacts);
What level of detail and accuracy is required (and your
stakeholders expect) from the outcome indicators and
evaluation; and
What data already exists that might be used as part of the postimplementation
evaluation.
The focus of post-implementation evaluation: individuals or systems?
Impacts and associated outcomes can be assessed at two main levels: the individual and the system levels. It is generally necessary
to evaluate at both levels, but where the emphasis is placed will have a major impact on the types of outcome indicator selected, the
data collection and analysis techniques, the time-frames, costs, etc.
Individual level impacts measure the change of behaviour
of individuals or similar groups of persons. At this level it might
be necessary to have a 'target' group and a 'control' group, to
establish whether the observed changes in behaviour are really
due to implementing the project, or to some other external
factors. This usually requires surveys and interviews. Members
of the control group should have similar characteristics to the
target group, and the data should be collected at the same time
as for the target group.
In order to identify what has changed and why, it is important to
recognise that there are different stages in the behavioural
change process, each of which may require monitoring, under
certain circumstances. An example of such a behaviour change
model from the European research project TAPESTRY is
illustrated on the opposite page.
At the system level, aggregate changes in the use of the
transport system, or project impacts on, say, the environment
are the centre of attention. At this level, some wider outcomes
of a project can be evaluated, such as corridor accident
reductions or increases in city centre economic activity. Much
system level data is already collected by different agencies,
though often large amounts of data have to be handled (e.g.
continuous traffic counts), and judgements will need to be
made about how to sample from or compress the data. Hence,
data analysis and interpretation will be very different from that
at the individual level, and requires different skills. Again, the
question has to be addressed as to whether the observed
changes or effects were caused by the project or other
external factors or measures.
PROJECT MONITORING
Resources
(Input Indicators)
Activities / Actions taken
(Process Indicators )
Output
(Output Indicators )
Outcome Monitoring
and Evaluation
76