GAO-12-208G, Designing Evaluations: 2012 Revision

More documents

Recommendations

Info

Preface We welcome your comments on this paper. Please address them to me at kingsburyn@gao.gov. Nancy R. Kingsbury, Ph.D. Managing Director Applied Research and Methods Page 2 GAO-12-208G
Chapter 1: The Importance of Evaluation Design Chapter 1: The Importance of Evaluation Design What Is a Program Evaluation? A program evaluation is a systematic study using research methods to collect and analyze data to assess how well a program is working and why. Evaluations answer specific questions about program performance and may focus on assessing program operations or results. Evaluation results may be used to assess a program’s effectiveness, identify how to improve performance, or guide resource allocation. There is no standard government definition of “program.” A program can be defined in various ways for budgeting and policy-making purposes. Whether a program is defined as an activity, project, function, or policy, it must have an identifiable purpose or set of objectives if an evaluator is to assess how well the purpose or objectives are met. Evaluations may also assess whether a program had unintended (perhaps undesirable) outcomes. An evaluation can assess an entire program or focus on an initiative within a program. Although evaluation of a federal program typically examines a broader range of activities than a single project, agencies may evaluate individual projects to seek to identify effective practices or interventions. Program evaluation is closely related to performance measurement and reporting. Performance measurement is the systematic ongoing monitoring and reporting of program accomplishments, particularly progress toward preestablished goals or standards. Performance measures or indicators may address program staffing and resources (or inputs), the type or level of program activities conducted (or process), the direct products or services delivered by a program (or outputs), or the results of those products and services (or outcomes) (GAO 2011). A program evaluation analyzes performance measures to assess the achievement of performance objectives but typically examines those achievements in the context of other aspects of program performance or in the context in which the program operates. Program evaluations may analyze relationships between program settings and services to learn how to improve program performance or to ascertain whether program activities have resulted in the desired benefits for program participants or the general public. Some evaluations attempt to isolate the causal impacts of programs from other influences on outcomes, whereas performance measurement typically does not. Evaluations have been used to supplement performance reporting by measuring results that are too difficult or expensive to assess annually or by exploring why performance goals were not met. (For examples, see GAO 2000.) Page 3 GAO-12-208G
Page 1 and 2: GAO January 2012 GAO-12-208G United
Page 3 and 4: Program Evaluation Standards, Joint
Page 5: Preface Designing Evaluations Prefa
Page 9 and 10: Who Conducts Evaluations? Chapter 1
Page 11 and 12: Five Key Steps to an Evaluation Des
Page 13 and 14: Chapter 1: The Importance of Evalua
Page 15 and 16: Figure 1: Sample Program Logic Mode
Page 17 and 18: Clarify the Issue Chapter 2: Defini
Page 19 and 20: Chapter 2: Defining the Evaluation
Page 21 and 22: Chapter 2: Defining the Evaluation
Page 23 and 24: Chapter 3: The Process of Selecting
Page 25 and 26: Collect Background Information Cons
Page 27 and 28: Plan Original Data Collection Chapt
Page 29 and 30: Select Evaluative Criteria Chapter
Page 31 and 32: Pilot Test Data Collection and Anal
Page 33 and 34: For More Information On sampling ap
Page 35 and 36: Chapter 4: Designs for Assessing Pr
Page 37 and 38: Assessing Variation in Implementati
Page 43 and 44: Typical Designs for Drawing Causal
Page 45 and 46: Limited Applicability of Randomized
Page 47 and 48: Difficulties in Conducting Comparis
Page 49 and 50: Comprehensive Evaluations Explore B
Page 55 and 56: Recode State Data into a Common For
Page 57 and 58:
Conduct an After-Action or Critical
Page 59 and 60:
External Expert Portfolio Review Bi
Page 61 and 62:
Conduct Effectiveness Evaluations i
Page 63 and 64:
Challenge: Isolating Impact When Se
Page 65 and 66:
Test Hypothesized Relationships bet
Page 67 and 68:
Chapter 5: Approaches to Selected M
Page 69 and 70:
American Evaluation Association Gui
Page 71 and 72:
Other Papers in This Series Other P
show all

GAO-12-208G, Designing Evaluations: 2012 Revision

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?