TRIPLE HELIX noms.pmd

P-027Sectoral Funds: Methodological Framework for Evaluation of Scientific andTechnological Development Actions at the National Nuclear EnergyCommission - CNEN/BrazilPierre Ohayon, Universidade Federal do Rio de Janeiro - UFRJ, BrazilIvan Pedro Salati de Almeida, Francisco Rondinelli, Comissão Nacional de Energia Nuclear - CNEN, BrazilGerson Rosenberg, Fundação Oswaldo Cruz - FIOCRUZ, BrazilSince the first experiences of government programs evaluation in Brazil, the improvement in development and implementation ofScience and Technology (S&T) evaluation systems has not been very significant, despite the importance of this theme. Thisstudy proposes an evaluation system for the Energy Sectoral Fund concerning the National Nuclear Energy Commission /Brazil (ESF-CNEN). The specific objectives of the system are: (i) Measuring implementation levels of the ESF-CNEN objectivesand goals and also the adherence to international trends for scientific and technological research, (ii) Screening and analyzingrelevant results regarding scientific and technological development policy, embodied in Sectoral funds objectives and operatingsponsored projects institution goals, (iii) Recording all possible differences between expected and actual results, includinganalysis of its causes and implications, and (iv) Evaluating direct impacts of ESF-CNEN (related to its more immediate goals)and indirect impacts of sponsored projects on policies and institutions involved.The setting up of Sectoral Funds (Pereira, 2005) by the Brazilian Government in 1999 allowed the expansion of resourcesallocated for the Science, Technology and Innovation system (ST&I), strengthening the scientific and technological basis forindustry in specific sectors and improving integrated management among different sponsor agencies. The main sponsors of theST&I system are FINEP – the national technological development and innovation Agency - and CNPq – the national council forscientific and technological research, both from MCT - Brazilian Ministry of Science and Technology.This study focuses on the use of Sectoral Funds applied to Brazilian nuclear area. The motivation to develop a methodologicalframework for evaluating science and technology actions for the nuclear area is caused, on one side, by the new Governmentpriorities in this field and, on other side, the experience of staff members at CNEN engaged in this research.There is not yet in Brazil an "Observatory" in charge to systematically design and produce indicators which describe thescientific and technological activities within country, spreading later to the overall participants (“actors”) of the Brazilian researchand technological development system. Indeed, "there is still not a tradition in Brazil for the production of indicators and data inthis area" (FAPESP, 2002, p. 3).The role of evaluation is gaining importance as a management tool, especially for decision-making, including the establishmentof public policies. In this way, the study provides subsidies to CNEN (institutional operator), FINEP and CNPq (national agenciesresponsible for the Sectoral Funds management) on an evaluation model and especially in its logical evaluation framework.The Evaluation Model proposed for ESF-CNEN provides subsidies for evaluating ESF-CNEN, considering the standards set bythe Sectoral Fund, and undertakes analysis of aspects related to efficiency, efficacy and effectiveness of ESF-CNEN and itssocial and economic impact.The evaluation of EFS-CNEN is understood as a systematic collection of information about activities, characteristics andoutcomes of different actions, aiming to reduce uncertainty and improve effectiveness of these actions and decision-makingprocesses. It is important to consider that the implementation of different projects should maintain consistency with the overallgoals of ESF-CNEN, requiring up continuous assessments (Clark, 1999, p. 14 apud Lima, 2004).The suggested model should a priori facilitate the construction of an evaluation instrument able to consider the most relevantaspects of ESS-CNEN and its social impacts. Projects and programs evaluations traditionally focus on issues such as criteriaof efficiency, efficacy, effectiveness and quality of management (Ohayon & Rosenberg, 2007).Besides that, this model focus on the impact of technological programs following the more recent trends of adapting themethodologies elaborated in developed countries to the Brazilian environment.The model also sees ESF-CNEN as being formed by four functional subsystems: (1) Political, Strategic and Normative Subsystem;(2) Organizational Subsystem: (3) Technical and Scientific and Economic Subsystem; and, (4) Resources Allocation andManagement Subsystem. These four ESF-CNEN subsystems interact among themselves, in all levels of their programmaticaction (annual or multi-annual plans), influence the behavior of all parts involved (institution and teams under the projects) andimpact the external environment, in their respective domains.In the assessment of each subsystem, besides comparing the ESF-CNEN performance indicators with the original planning,the study shows how important is to analyze them in a wider context. In the Technical-Scientific and Economic Subsystem, forexample, it is important to know the results obtained with training of human resources and compare it with other initiatives inthis area, like those taken with the institution budget. This can measure how important the resources from ESF-CNEN were infact to the organization.Performance Indicators are established for each important process in the subsystems, such as: (i) Inputs Indicators; (ii)Outputs Indicators (iii) Impact Indicators (iv) Economic Indicators. To evaluate the results of ESF-CNEN it is necessary toestablish a set of analysis approaches. The objective is to define specific aspects to be assessed using the indicators andcriteria associated. This is necessary since the word “evaluation” can be interpreted in several different ways and meanings(GEISLER, 2000). The statement of a set of analyses approaches can define clearly the objective of the evaluation. Theanalysis approaches can be related to one specific subsystem or to several subsystems.Finally, this article brings about different recommendations to improve evaluation for science and technology programs.Madrid, October 20, 21 & 22 - 2010246