Towards a Metrics Implementation Plan - UCAR Finance ...

More documents

Recommendations

Info

Table 3 of the UCAR MetricsCommittee Report, reproduced inpart in the text box at right, showsgeneral metrics that apply to morethan one functional category ofUCAR activities and some keymetrics used in the ten functionalcategories of UCAR activities. All11 major metrics are listed in theUCAR Metrics Committee Report’sTable 3.The UCAR metrics committeerecommended that both metrics andindependent and knowledgeablepeer reviews should be used toevaluate UCAR’s process, as wellas its progress toward its goals.Table 3*Metrics to characterize the productivity,quality, impact, and efficiency of UCARprogramsGeneral—Apply to more than one category of UCARactivitiesKey metrics used in ten categories of UCAR activities1.0 Science2.0 Scientific visitors and postdoctoral fellows3.0 Computing facilities4.0 Community models5.0 Observing facilities6.0 Data services7.0 Intellectual property and technology transfer8.0 Administrative support9.0 Education, outreach, and training10.0 Communications and advocacy*Taken directly (as Table 3) from “Measuring the Productivity,Quality, and Impact of UCAR Programs” (UCAR 2005).3.4 Notes on the UCAR Metrics Committee ReportWhen UCAR’s metrics committee members wrote the UCAR Metrics Committee Report, theydid so without the benefit of the UCAR*2020 Strategic Plan, which had not yet been written. Asa result, the UCAR Metrics Committee Report and the UCAR Strategic Plan are twoindependent documents that stand alone.As discussed in the Section 3.2, the NRC document strongly recommends including five types ofmetrics in a robust set: (1) process metrics, (2) input metrics, (3) output metrics, (4) outcomemetrics, and (5) impact metrics. The Team performed a exercise to evaluate the metricsidentified in the UCAR Metrics Committee Report into the five types of metrics. Theseassignments are shown in Appendix A.The UCAR Metrics Committee Report determined what metrics should be applied to UCAR's 10functional categories. We matched those metric categories to the UCAR Strategic Plan's 7strategic goals. Multiple metrics could apply to multiple goals. We handled this challenge byselecting a “primary” metric category, or categories, to assign to each goal. This approachreduces the complexity of measuring progress towards a specific goal.Appendix B, Table B-1 matches functional metric categories to UCAR strategic goals. Theindividual metrics of each category are then listed showing the item to measure, the measuringunits, and the focus, or importance, of the measure. The focus of a measure indicates whether ornot the metric should be used for measuring UCAR's production levels, quality of work, or theimpact on external entities or events. Some metrics have more than one focus.The current list of UCAR metrics can be grouped into three out of five categories—OutputMetrics, Outcome Metrics, and Impact Metrics.6
Strong leadership in UCAR management and consequently in the organization’s strategic planspartially fulfills the requirements of the Process Metrics. Additional Process Metrics for settingpriorities and allocating resources among elements of the program need to be created.It is interesting to note that the proposed UCAR metrics are dominated by Output Metrics, whichmeasure products and services.Input metrics, which measure tangible quantities needed to achieve a goal, are not considered inthe current list. Input metrics include resources to implement and sustain research, humancapital, measurement systems, prediction models, data services, and infrastructure. This may bean inherent weakness in the UCAR Metrics Committee Report because results of output metrics(good, bad, or ugly) cannot be explained if there is no way of knowing that enough resourceswere spent on achieving a desired goal. As a result, this lack of input metrics may limit theability to explain output metrics.We can see, then, that the current list of UCAR metrics needs to be augmented to fill the gaps inProcess and Input metrics.4 Outline of Metrics Implementation Plan Process for UCAR4.1 Process IntroductionIn this section, we outline a potential process for UCAR to follow in developing an MIP. Wedeveloped this process based on our experience with and knowledge of UCAR/NCAR/UOP andour vision of what the MIP should include at the end of the process. We offer this as a foundationon which to build a final MIP development process that can be altered, supplemented, andrevised as needed based on the organization’s objectives and goals and the resources available.We are as explicit as possible in stating the assumptions and decisions that need to be made indeveloping such a process.The NRC document 3 states that “A good strategic plan must precede the development ofmetrics.” Our process assumes the UCAR Strategic Plan, currently under development, as itsfoundation. Integration with the completion and adoption of the UCAR Strategic Plan is aprerequisite for final deployment of the MIP.In suggesting this process we view thus the MIP as an integral component of overall UCARstrategic planning and UCAR/NCAR/UOP management focus. We feel that these metrics willbecome a crucial tool in management, program evaluation, and budget decision making, as wellas in institutional culture.As we developed this process, we assumed a “top-down” approach. By this we mean that UCARmanagement will strongly support this effort, adopt an MIP development process, and entrainother units (NCAR, UOP) into this commitment. If this effort were to involve only developmentof metrics for UCAR, the approach and effort would likely be significantly different thansuggested here. We also suspect that, if NCAR and UOP are not fully on board with this effort,the exercise will fail to meet stakeholder requirements (e.g., NSF’s desire to see metricsindicative of the return on its investment).3 Executive Summary, p. 37
Page 1 and 2: Toward a Metrics Implementation Pla
Page 3 and 4: 1 IntroductionThis document’s obj
Page 5 and 6: similar to those associated with im
Page 7: 3.2 National Research Council’s
Page 11: Step1: Commitment by UCAR/NCAR/UOP
Page 16 and 17: its particular goal. We advise the
Page 18 and 19: • UCAR/NCAR/UOP Management/Staff
Page 20 and 21: Figure 2. Proposed MIP process time
Page 22 and 23: 5.0 Observing facilities1. Number o
Page 24 and 25: Strategic GoalTable B-1. How UCAR M
Page 26 and 27: Strategic GoalTable B-1. How UCAR M
Page 28: Strategic GoalTable B-1. How UCAR M

Towards a Metrics Implementation Plan - UCAR Finance ...

Create successful ePaper yourself

Delete template?

Save as template?