The Evaluation of 'Behavioural Additionality' - IWT

More documents

Recommendations

Info

CHAPTER 3 > Conceptual and Empirical Challenges of Evaluating the Effectiveness of Innovation Policies with ‘Behavioural Additionality’ . makers to explicitly target behavioural changes in the design of policy instruments. Most notorious examples are programmes aiming at fostering co-operation between the actors in the innovation system in order to increase the productivity of the knowledge investments through spillovers. The inability of present monitoring and evaluation methods to adequately assess the effectiveness of the new system oriented policies are an important bottleneck for their generalisation and therefore for performance improvement of innovation systems. System policies capitalise on the synergistic character of the innovation process, while traditional policies (and evaluation methods) only perceive sectoral policy actions towards individual agents on project level. Therefore increased efforts to strengthen the capabilities for evaluation are needed. Because of the recent and still immature nature of the new approach on ‘behavioural additionality’ efforts for conceptual clarification and for standardisation of empirical research methods can contribute to form an international community of practice that may foster the capacity for international policy learning. The work of the OECD to establish a framework for research and for exchange of good practices on this subject is of great importance for national innovation policies. The aim of this document is to contribute to the clarification of the research agenda, based on the experience of IWT Flanders in setting-up an evaluation method for its R&D-subsidies. The following chapter starts from the experience of evaluation of bottom-up innovation stimulation in Flanders. Additionality is embodied in several specific selection requirements. This practice expresses a growing understanding of the complexity of innovation processes but evaluation of the effectiveness of these schemes by their stated objectives cannot fully cover the new systemic perspective of the role of government. Behavioural additionality appears as a ‘bridging concept’ in the evolution towards third generation innovation policies that takes account of the interaction effects of public and private strategies. > The next two chapters discuss the opportunities and limits of this new approach on the level of the concept and its measurement. The main message of introducing behavioural additionality is the focus on learning, including policy learning, to enhance innovation performance. The last chapter briefly describes the complementary components of the comprehensive evaluation strategy that is needed to implement this evaluation perspective and subsequent new role for public support agencies in the innovation system. 2. EVALUATION AND ‘ADDITIONALITY’ IN FLANDERS 2.1. ACTUAL PRACTICES OF EVALUATION IWT is the agency of the Flemish government for the implementation of the Flemish innovation policy but has also important tasks in policy preparation. It was established in 1992 to give shape to the new competences in science and technology that were transferred from the federal to the regional governments in Belgium and controlled a budget of 235 million euro in 2003 (40% of the overall Flemish S&T budget). IWT plays an important role in the new Flemish innovation system. IWT fulfils different tasks; evaluation therefore has many dimensions and levels. Although the question of efficiency (use of given resources) is important to IWT management, is subordinated to the question of effectiveness (attainment of given goals) in the evaluation of stimulation activities such as R&D subsidies. The first question implies the improvement of control and capacity management; the second is oriented towards evaluating desirable outcomes, which is the final policy relevance of the intervention. Do IWT actions make a difference in the improvement of innovation performance Evaluation culture in Flanders is slowly developing, largely as ‘learning by doing’. Evaluation of effectiveness is not supported with specialised, scientifically based and professionally managed evaluation departments. To explain this learning 58
CHAPTER 3 > Conceptual and Empirical Challenges of Evaluating the Effectiveness of Innovation Policies with ‘Behavioural Additionality’ . IWT-STUDIES > >> 48 process in evaluation, and the integration of the additionality dimension, we distinguish the different levels of evaluation practice. Most common in IWT is the project evaluation. IWT is the one-stop-shop for subsidies to industrial R&D in Flanders. The general feature of the subsidy scheme is its bottomup character: it is a permanently open and non-thematic scheme. IWT has a well-developed set of procedures for project evaluation, based on internal and external referees, to evaluate the ex-ante effectiveness of the project proposals (ex-post evaluation is starting up). Initially evaluation criteria were heavily focussed on the scientific qualities and technological risks of the project, but gradually the economic dimension (‘utilisation’) became equally important, reflecting the shift from an R&D towards an innovation policy focus. This economic evaluation doesn’t only concern the financial feasibility of the project or the commercial prospects for the innovating firm but also the economic return ‘for Flanders’. The most important economic criterion is that the project must be able to generate an economic value added that is at least ten times the value of the subsidy. During the evolution of this IWT evaluation practice also the ‘societal’ qualities of the project – mainly concerning environmental sustainable development - became part of the evaluation and selection procedure, although not on the same level as the two main axes. But because of the formal policy objective to increase the number of projects that support sustainable development, the latter criteria were formally separated from the standard project evaluation in a separate arrangement for ‘sustainable technological development’. This arrangement applies across the existing subsidy schemes and implies an additional evaluation of candidate projects on criteria of ‘eco-efficiency’. The evaluation gives access to extra support in the form of a priority ranking in the subsidy scheme concerned and of a financial bonus of 10% on the project budget. This evolution illustrates that project evaluation in Flanders is closely linked to general policy criteria in a bottom-up innovation policy design. IWT also has an important stake in programme evaluations, because the ‘self-evaluation’ by the management (up to now mostly ex-post evaluation) is an important input for the policy (re)design. Programmes (or support schemes) don’t have a thematic focus but are designed on the basis of characteristics of specific target groups. The ‘SME Innovation Programme’ is also a bottom-up scheme but supports in addition to research projects also feasibility studies and allows submitting a broader range of innovation costs to lower the threshold for innovation in SMEs. Another important programme is the ‘Strategic Basic Research’ Programme that is also non-thematic but organised by a yearly call to form consortia between universities and companies for longer term projects that are strategically important for Flanders from an economic and/or social point of view (also non-technological research is included). Programme evaluation is closely linked to the achievement of these non-technological objectives. Innovation policy has developed a complete set of support schemes along the ‘innovation chain’ that are put available for bottom-up initiative. But it is felt that programme design has to fit the different leverage needs and leverage potentials of various types of innovation activities and actors. Evaluation criteria try to discriminate access on that general level. What is still lacking in Flanders is an overall policy evaluation that takes into account the policy mix as a whole (the complementarity of the different programmes) and assesses the performance of the innovation system in an international perspective. This is a component of the organisation of the policy cycle that is not yet formalised, partly because the institutionalisation of the innovation system is not fully completed yet (a formal Ministry of Sciences and Innovation will be established in 2004). But the main reason is that the need for this was not urgently felt because of the strong immersion of policy objectives on the operational levels by means of differentiated ‘bottom-up’ programmes that give the initiative to the business and research actors. Formal top-down objectives of government for the innovation system were inexistent. 59
Page 1 and 2:
IWT-Studies IWT-Observatory Innovat
Page 3:
IWT-STUDIES > >> 48 CONTENTS FOREWO
Page 6 and 7:
FOREWORD Consult), building on thes
Page 8 and 9: CHAPTER 1 > Evaluation of Behaviour
Page 24 and 25: CHAPTER 2 > Measuring Additionality
Page 48 and 49: APPENDIX > Additionality Questionna
Page 57: Chapter 3 CONCEPTUAL AND EMPIRICAL
Page 61 and 62: CHAPTER 3 > Conceptual and Empirica
Page 71 and 72: REEDS VERSCHENEN BIJ HET IWT-OBSERV
Page 75 and 76: BIOGRAPHY LUKE GEORGHIOU Luke Georg
show all

The Evaluation of 'Behavioural Additionality' - IWT

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?