EVALUATION

AN INDEPENDENT EVALUATION OF THE LCNF
benefits where the results are identifiable and credible. For this evaluation work we
have defined ‘future benefits’ as being those that accrue between 1 April 2016 and 31
March 2031 (the end of the RIIO-ED2 period) 22 .
2.5.3.1 Analysing the data
Given the uncertainty associated with measuring the potential benefits associated with
innovation, we have discussed above, we have only monetised the potential benefits
where we believe the results are robust and credible and can be defended.
Following receipt of the information from the DNOs we have undertaken analysis and
validation of the results. Given the nature and stage of the LCNF projects there are no
independent sources of information available by which to verify the quantitative data. As a
result, our analysis has been undertaken based on a review of other DNO submissions,
for example this includes the reports published as part of the LCNF process and
benchmarking between comparable projects. Specifically this process has included:
• cross-checking the results against other publically available data, such as project
‘Close Down’ reports and bid submission, to make sure that the estimated benefits
had consistency;
• ensuring consistency within, and between, the DNO submissions. For example we
looked at LCNF projects with similar innovation initiative to check for any major
difference in the benefits estimated; and
• analysing the results to understand the different types of initiative being investigated
by the LCNF projects. In doing this we drew on the academic classification system
developed by the University of Strathclyde in its report on the LCNF 16 .
Finally, we spoke individually with each of the DNOs to resolve any questions we had on
the original information provided. During these discussions we resolved a number of
potential issues 23 and through the clarifications we were able increase the robustness of
the information provided. We are reasonably comfortable with the veracity of the data
provided by the DNOs – recognising that these numbers would change if the DNOs had
more time to undertake a more comprehensive assessment.
2.5.4 Defining a robust baseline
Essentially the quantitative element of this evaluation has attempted to identify any
‘additionality’ from the LCNF scheme. That is, what has been, or is expected to be,
delivered in addition to what might reasonably be expected to have happened anyway.
To understand this additionality we have established a ‘counterfactual’ position based on
what we would have expected to happen if the LCNF has not been implemented.
Establishing such a counterfactual is challenging, since by definition it cannot be observed
– it is what would have happened if the policy had not gone ahead. A strong evaluation is
one which is successful in isolating the effect of the policy from all other potential
influences, thereby producing a good estimate of the counterfactual.
22
23
This assessment period is, in general, shorter than the assessment period used by DNOs in
their project closedown reports.
For example we identified examples of double counting of benefits between projects and
differences in the potential benefits between the questionnaire response and other publically
available reports.
PÖYRY MANAGEMENT CONSULTING
October 2016
713_Poyry_Report_Evaluation_of_the_LCNF_FINAL_Oct_2016_v700.docx
19