public program evaluation - Askew School of Public Administration ...

PUBLIC PROGRAM EVALUATION
PAD5327, Spring 2013
Prof. Fran Berry
Class Hours: Wednesday 5:30-8:15 p.m.
Office Hours: Tuesday 3:00-5:00p.m. & Wednesday 3:00-5:00p.m. or by appointment
Office: 003 Bellamy Building, office phone--644-7603; fax 644-7617; cell 443-0061
e-mail: fberry@fsu.edu
I. COURSE OBJECTIVES
This class is designed for the in-service and pre-service professional public manager and
policy analyst, as well as for doctoral students who will teach program evaluation. The
course is one of the course requirements for the MPA student taking the Policy Analysis
concentration in the MPA program. At the end of the semester, each student in the course
should be able to meet each of the following seven learning objectives:
Learning Objective One: Be able to describe the different types of program evaluations,
and select and apply each type of evaluation for an appropriate purpose in the program
environment in which evaluations are conducted;
Learning Objective Two: Know how to identify stakeholders for program evaluations,
and engage them in useful, economical ways for the evaluation, as well as understanding
the roles of evaluators in making evaluations useful and relevant to the stakeholders;
Learning Objective Three: Demonstrate familiarity with the primary types of research
designs used in program evaluations;
Learning Objective Four: Introduce the practitioner with the primary methods (including
both quantitative and qualitative methods) for conducting evaluations, both to enable the
practitioner to conduct sound program evaluations and to critique evaluations they
oversee or read;
Objective Five: Develop and demonstrate skills in constructing logic models and program
performance measures, collecting data, and writing succinct reports on evaluations;
Learning Objective Six: Discuss the actual and ideal roles of the evaluator in the
executive and legislative policy processes, including potential conflicts of interest and
ethical issues evaluators face; and
Learning Objective Seven: Learn how to successfully manage a program evaluation
project to promote full utilization of the evaluation into the policy process, and to have
the experience of actually conducting an evaluation of a public program.

II. Course Requirements
Grading:
Evaluation Project: 30%
Short Project One: 15%
Short Project Two: 15%
Mid-term exam: 30%
Class Participation: 10%
The class will be interactive, thus it is important that you have completed the readings for
each class session and come prepared to discuss the material. Students will be asked to
lead a discussion or group facilitation or case study review on a topic related to the
overall class focus, mostly using materials I provide them. I will be assigning additional
short evaluations or cases throughout the semester that are not posted on the syllabus.
There will be two short written projects, and these will be detailed in a handout, which
will consist of a short written report in a briefing format or a critical review of one types
of evaluation theory or approach. There will be an in-class mid-term exams, which will
consist of short answer and/or essay questions, or evaluation critiques which will
integrate major issues from the course. The major written product for the class is an
evaluation project design, which will be described in detail in a written handout and will
count 30% of your class grade. Brief presentations on your evaluation project will be
presented in class during April. Class members (in groups of two) will be asked to make
presentations of about 15 minutes on a topic related to the night’s readings and themes,
choosing from a list compiled by Dr. Berry. The topics, readings and a signup for the
semester will be presented early in the semester.
Please let me know if you will not be in class, as I do keep track of attendance. The
student honor code is enforced in the class; if you are unfamiliar with it, please refer to
the Student Handbook. I expect that the short written project and the final exam will be
your own work alone, and not conducted with help from any other person.
If you see materials in the popular media or in your work place on program evaluation
that you think are pertinent, please bring them in to share with me and the class.
Note: For items submitted to me, please no binders; a staple in the top left-hand corner is
very functional. Please send me an electronic version of each hard copy assignment you
submit to me. Even with great care, sometimes things get lost.
ACADEMIC CONDUCT
Students are expected to uphold the Academic Honor Code published in The
Florida State University Bulletin and the Student Handbook. The Academic Honor
System of The Florida State University is based on the premise that each student has the
responsibility (1) to uphold the highest standards of academic integrity in the student's
own work, (2) to refuse to tolerate violations of academic integrity in the university
community, and (3) to foster a high sense of integrity and social responsibility on the part

of the university community. Please see the following web site for a complete
explanation of the Academic Honor Code. http://www.fsu.edu/Books/Student-
Handbook/codes/honor.html
Academic dishonesty includes, but is not limited to:
Plagiarism: quoting or paraphrasing the ideas or opinions of others without
appropriate attribution in text citations and a reference list. This includes
books, journal articles, conference presentations, published or unpublished
papers and web-based materials.
Fraud: submitting work that was not prepared by you, or which you have
previously submitted for another class.
Cheating: giving help to other students, or asking them for it, on the final
examination. The consequence of academic dishonesty is a grade of F on the
assignment in question
AMERICANS WITH DISABILITIES ACT
Students with disabilities needing academic accommodation should:
(1) register with and provide documentation to the Student Disability Resource Center;
and
(2) bring a letter to the instructor indicating the need for accommodation and what type.
This should be done by the second week of class. This syllabus and other class materials
are available in alternative format upon request. For more information about services
available to FSU students with disabilities, contact the:
Student Disability Resource Center
97 Woodward Avenue, South
108 Student Services Building
Florida State University
Tallahassee, FL 32306-4167
(850) 644-9566 (voice)
(850) 644-8504 (TDD)
sdrc@admin.fsu.edu
http://www.disabilitycenter.fsu.edu/
SYLLABUS CHANGE POLICY: This syllabus is a guide for the course and is subject
to change with advance notice. Additional readings will be added in the future.
III. COURSE TEXTS
The following text is required reading and can be purchased at the University Bookstore
or Bill's Bookstore, or online.
Evaluation in Organizations: A Systematic Approach to Enhancing
Learning, Performance and Change by Darlene Ruff-Eft and Hallie
Preskill, Basic Books, New York, New York, second edition, 2009.
Optional as background materials but not required in the class:
James Sanders, ed., The Program Evaluation Standards, third edition, Sage
Publications, Thousand Oaks, CA 2010.

Handbook of Practical Program Evaluation (Jossey Bass Nonprofit &
Public Management Series) by Joseph S. Wholey, Harry P. Hatry, and Kathryn E.
Newcomer, Third edition, 2010.
An excellent resource of materials on how to conduct community-based
evaluations is from the Community Toolbox on the University of Kansas website
which has a 46 chapter handbook on how to promote community health through
evaluation and analysis. The URL is: http://ctb.ku.edu/en/tablecontents/index.aspx
IV. CLASS SCHEDULE:
TOPICS AND ASSIGNED READINGS
January 9 th
INTRODUCTION AND COURSE OVERVIEW
January 16 th History and where Evaluations are Used;
Standards for Evaluation, Types and Purposes of Evaluation; Designing
Useful Evaluations
Russ-Eft and Preskill, Evaluation in Organizations, Chapter 1 Defining Evaluation and
Chapter 2 The Evolution of Evaluation
Topics to cover: Gain a brief overview of how program evaluations have developed
over time, and where they have been practiced; Understand the different purposes for
conducting program evaluations, what questions are asked, and how this can impact the
design and implementation of an evaluation.
January 23 rd The Principles and Practice of Causal Inference;
Using Logic Models; Performance Measurement
**Logic Models from the Community Toolbox, University of Kansas
Russ-Eft and Preskill, Evaluation in Organizations Chapter 5 Focusing the Evaluation
Topics to cover: How to establish causal inference and why it is important in an
evaluation research design; understanding logic models and how to develop them to
illustrate the causal linkages among activities, sub-objectives or short term outcomes, and
longer term outcomes; should all evaluations be causally directed Understanding how to
develop the evaluation’s rationale and purpose. Developing logic models that will be
useful to stakeholders, and help direct the measures used for the evaluation.

January 30 th Working with the Evaluation’s Stakeholders and Clients;
Roles of the Evaluator
**Wholey, Hatry and Newcomer, Chapter 2 Analyzing and Engaging Stakeholders
Topics to cover: Identifying and engaging stakeholders of the program you are
evaluating is a necessary and important part of the program evaluation process, both to
get their views on how the program is operating, and to find out what information and
criteria they believe the program should be assessed by. Considering the diverse roles
that evaluators can play in an evaluation, and the skills and professional duties that the
roles require. Considering choices about what type of evaluator you want to be.
February 6 th Selecting a Research Design: Process/Implementation
and Formative Evaluations; Time series evaluations.
Russ-Eft and Preskill, Chapter 6 Selecting an Evaluation Design from Evaluation in
Organizations,
**Tim Weston, “Formative Evaluation for Implementation” (BB)
**U.S. General Accountability Office, Medicaid Expansion: States’ Implementation of
the Patient Protection and Affordable Care Act, August 2012.
Topics to cover: Research designs are the logical method of obtaining data and using
analysis to determine the findings. Everyone should have a good familiarity with the
various research designs available to the program evaluator, and the logic of getting
comparative data for a control of comparison group, as well as gathering data before and
after treatment, if that is possible. Assessing whether a program or treatment is
implemented before/during your evaluation; understanding the impact that program
implementation has on program evaluation; learning the basic elements of how to
conduct a process evaluation and an evaluability assessment.
February 13 th
Research Designs: Experimental Designs and Random
Experiments
**Ann Solberg, “Community Posthospital Follow-up Services,” Evaluation Review vol.
7 (1) Feb. 1983. (BB)
**Jay Greene et.al, “School Choice in Milwaukee: A Randomized Experiment” from
Evaluation in Practice by Bingham and Felbinger, 2009. (BB)
Topics to cover: Properties of experiments; the logic of experimental design, including
the importance of random assignment ; threats to internal validity; external validity;
participation in programs, including bias and program coverage

February 20 th Getting Customer/Client Input through surveys ;
Sampling for Surveys
Russ-Eft and Preskill, Evaluation in Organizations Chapter 10 Surveys and
Questionnaires;; Chapter 12 Sampling
Topics to cover: How to construct and implement great surveys; sampling methods and
when and why to use them.
February 20 th
First Project Due
February 27 th Quasi-Experimental Designs: Interrupted
Time Series; Developing Measures and Collecting Data
Russ-Eft and Preskill, Evaluation in Organizations Chapter 7 Choosing Data Collection
methods; Chapter 8 Archival Data; Chapter 9 Observation
**Ken Meier, Executive Reorganization of Government: Impact on Employment and
Expenditures, AJPS, 1980.
Optional for short presentations:
Wholey, Hatry and Newcomer; Chapter 13 Using Trained Observer Ratings; Chapter 14,
Collecting Data in the Field; Chapter 15, Using the Internet, Chapter 16 Focus Group
Interviewing, and Chapter 17 Using Stories in Evaluation
Optional:
Kickham, Kenneth and David Ford, Are State Marriage Initiatives Having an Effect An
Initial Exploration of the Impact on Divorce and Childhood Poverty Rates”, in
Public Administration Review, September/October 2009, pages 846-854. (BB)
Topics to cover: Quasi-Experimental Designs, and how interrupted time-series can be
sued to rule out alternative explanations. How to collect data from multiple sources.
Focus groups and field data are also widely used for many purposes to get in-depth view
or people or clients or data from client files.
March 6 th
Analyzing Data, Quantitative and Qualitative
Approaches
Russ-Eft and Preskill, Evaluation in Organizations; Chapter 11 Individual and Focus
Group Interviews; Chapter 13 Analyzing Evaluation Data
Topics to cover: A broad overview of most frequently used methods for data analysis
and interpretation within program evaluation fields, and the issues we consider in
choosing the best methods of data analysis. A brief overview of the world view behind
qualitative approaches to program evaluation and how they differ from quantitative
approaches.

March 13 th
NO CLASS SPRING BREAK
March 15 Second Project Due
March 20 th Linking Performance Measurement and
Management to Improving Programs, and Gathering Data for
Ongoing Program Assessment
**Wholey, Hatry and Newcomer; Chapter 5 Performance Measurement (BB)
**Robert Kaplan, Presentation on the Balanced Scorecard:
Performance Management in Public Sector Organizations (BB)
**Summary Information on the Sterling Award Criteria (BB)
**OPPAGA, (December 2003) State Faces Challenges to improving community Public
Health in Florida (No. 03-71.) and also four page update from June 2006—“Steps
Have been Taken to Improve Community Public Health Infrastructure
Throughout Florida” (BB)
“Performance Measurement and Benchmarking” Chapter 4 from Evaluation in Practice
by Bingham and Felbinger (BB)
Topics to cover: Using data collected for planning, budgeting and annual reports for
program assessment and evaluation, including frameworks of the Balanced Scorecard,
and the Sterling Criteria. How can managers integrate program evaluation into other
ongoing management duties
March 27 th
Mid-term Exam (in-class)
April 3 rd Analyzing Data and Communicating Your
Findings; The Politics and Ethics of Evaluations
Russ-Eft and Preskill, Evaluation in Organizations Chapter 4
Russ-Eft and Preskill, Evaluation in Organizations Chapter 13 Evaluating Evaluation
Data; and Chapter 14 Communicating and Reporting Evaluation Activities and Findings;
**Program Evaluation Case
Topics to cover: A broad overview of most frequently used methods for data analysis
and interpretation within program evaluation fields.

APRIL 10 TH Managing Evaluations; Institutional Review Boards
(IRBs); The Request for proposal (RFP); Managing An Evaluation;
Implementing And Evaluating an Evaluation
Russ-Eft and Preskill, Evaluation in Organizations Chapter Chapter 15 Planning,
Managing and Budgeting the Evaluation; Chapter 16 Evaluating the Evaluation, and
Chapter 17 Strategies for Implementing Evaluations in Organizations
**Will assign an evaluation proposal (RFP) to read
Topics to cover: basic introduction to writing an evaluation proposal; sources of
information for tracking evaluation requests for proposals (RFPs), roles for contract
management, gantt charts and project management; developing an evaluation budget.
April 17 th
Cost Benefit and Cost Effectiveness Evaluations
*Bingham and Felbinger, Chapter 13 and 14 (BB)
**FL OPPAGA, Intermediate Sanctions for Non-Violent Offenders Could
Produce Savings, March 2010.
The Program Evaluation Standards on cost benefit analysis (BB)
Topics to cover: role of cost benefit and cost effectiveness designs in policy analysis and
program evaluation; issues to consider in conducting cost benefit and cost effectiveness
designs and methodology for each.
April 24 th
April 24 th
Evaluation Projects Presented in Class
Evaluation Project Paper DUE
April 29-May3, 2013
THIS CLASS)
Final Exam Week (NO FINAL EXAM FOR
May 3 Graduation
Websites of Evaluation Materials and Resources
Americhan Evaluation Association (AEA), Magnolia, AR, http://www.eval.org. AEA is an international
professional association of evaluators devoted to the application and exploration of program evaluation,
personnel evaluation, technology, and many other forms of evaluation.
Campbell-Kibler Associates, Inc., Groton, MA, http://www.campbell-kibler.com. Campbell-Kibler Associates
is a private evaluation consulting firm with free resources available on their web site.
Horizon Research, Inc., Chapel Hill, NC, http://www.horizon-research.com. A private evaluation consulting
firm with free resources available on the web site.

The Evaluation Center at Western University, Kalamazoo, MI, http://www.wmich.edu/evalctr/index.html. The
Evaluation Center’s mission is to provide national and international leadership for advancing the theory and
practice of program, personnel, and student/constituent evaluation, as applied primarily to education and
human services.
Survey Suite at the University of Virginia, http://intercom.virginia.edu/surveysuite. This site allows you to
develop online surveys for your evaluation, gives you a secure access to allow participants to complete
surveys, and allows you to download the data as a statistical summary and/or a Microsoft Excel file.
Web Publications
Bond, S. L., Boyd, S. E., & Rapp, K. A. (1997). Taking Stock: A Practical Guide to Evaluating Your Own
Programs. Chapel Hill, NC: Horizon Research. [ www.horizon-research.com/publications/stock.pdf ]
Frechtling, J. and Sharp. L. (Eds) (1997.) User-Friendly Handbook for Mixed Methods Evaluation.
Washington, DC: National Science Foundation. [ www.ehr.nsf.gov/ehr/rec/pubs/nsf97-153/start.htm ]
Stevens, F., Lawrenz, L. & Sharp, L. (1993). User-Friendly Handbook for Project Evaluation: Science,
Mathematics, Engineering and Technology Education. Washington, DC: National Science Foundation.
[ http://www.nsf.gov/pubs/2002/nsf02057/start.htm]
Print Publications
Berdie, D. R. & Anderson, J.F. (1974). Questionnaires: Design and Use. New Jersey: Scarecrow Press.
Berk, R. A. & Rossi, P. H. (1990). Thinking About Program Evaluation. Newbury Park, CA: Sage.
Baker, T. L. (1998). Doing Social Research, 3 rd Ed. New York, NY: McGraw Hill.
Chelimsky, E. & Shadish, W. R. (1997). Evaluation for the 21 st Century: A Handbook. Newbury Park, CA:
Sage.
Converse, J. M. & Presser, S. (1986). Survey Questions: Handcrafting The Standardized
Questionnaire. Sage University Papers 63: Quantitative Applications in the Social Sciences.
Beverly Hills, CA: Sage.
Guba and Lincoln, Fourth Generation Evaluation
Davis, B. G. & Humphreys, S. (1993). Evaluating Intervention Programs. New York, NY: Teachers College
Press.
DeVillis, R. F. (1991). Scale Development: Theory and Applications. Newbury Park, CA: Sage.
Fink, A. & Kosecoff, J. (1985). How to Conduct Surveys: A Step-by-Step Guide. Newbury Park, CA: Sage.
Fitz-Gibbon, C. T. & Morris, L. L. (1987). How to Analyze Data. Newbury Park, CA: Sage.
Fitz-Gibbon, C. T. & Morris, L. L. (1987). How to Design a Program Evaluation. Newbury Park, CA: Sage.
Freeman, H. E., Rossi, P. H. & Sandefur, G. D. (1993). Workshop for Evaluation 5: A Systematic Approach.
Newbury Park, CA: Sage.
Glebocki, J. & Lancaster, D. (1984). In Search of The Wild Hypothesis. Arvada, CO: Anderson-Bell.
Gredler, M. E. (1995). Program Evaluation. Englewood Cliffs, NJ: Prentice Hall.

Greenbaum, T. L. (1997). The Handbook for Focus Group Research. New York, NY: MacMillan.
Henderson, M. E., & Morris, L. L. (1987). How to Measure Attitudes. Newbury Park, CA: Sage.
Herman, J. L., Morris, L. L. & Fitz-Gibbon, C. T. (1987). Evaluator’s Handbook. Newbury Park, CA: Sage.
Klecka, W. R. (1980). Discriminant Analysis. Sage University Papers 19: Quantitative Applications in the
Social Sciences. Beverly Hills, CA: Sage.
King, J. A., Morris, L. L. & Fitz-Gibbon, C. T. (1987). How to Access Program Implementation. Newbury
Park, CA: Sage.
Krueger, R. A. (1998). Analyzing and Reporting Focus Group Results. Newbury Park, CA: Sage.
Krueger, R.A. (1998). Developing Questions for Focus Groups, Vol.3. Newbury Park, CA: Sage.
Krueger, R. A. & Casey, M. A. (2000). Focus Groups: A Practical Guide to Applied Research. Newbury Park,
CA: Sage.
Lee, E. S., Forthofer, R. N. & Lorimore, R. J. (1989). Analyzing Complex Survey Data. Sage University
Papers 71: Quantitative Applications in the Social Sciences. Beverly Hills, CA: Sage.
Lofland, J. & Lofland, L. H. (1995). Analyzing Social Settings: A Guide to Qualitative Observation and
Analysis, 3 rd Ed. Belmont, CA: Wadsworth.
Miller, D. C. (1991). Handbook of Research Design and Social Measurement, 5 th Ed. Newbury Park, CA:
Sage.
Mohr, L.B. (1990). Understanding Significance Testing. Sage University Papers 73: Quantitative
Applications in the Social Sciences. Beverly Hills, CA: Sage.
Morris, L. L., Fitz-Gibbon, C. T. & Freeman, M.E. (1987). How to Communicate Evaluation Findings.
Newbury Park, CA: Sage.
Osterlind, S. J. (1983). Test Item Bias. Sage University Papers 30: Quantitative Applications in the Social
Sciences. Beverly Hills, CA: Sage.
Patton, M. Q. (1990). Qualitative Evaluation and Research Methods, 2 nd Ed. Newbury Park, CA: Sage.
Patton, M. Q. (1987). How to Use Qualitative Methods in Evaluation. Newbury Park, CA: Sage.
Rossi, P.H., Freeman, H.E. & Lipsey, M.W. (1999). Evaluation: A Systematic Approach, 6 th Ed. Newbury
Park, CA: Sage.
Schroeder, L. D., Sjoquist, D. L. & Stephan, P. E. (1986). Understanding Regression Analysis: An
Introductory Guide. Sage University Papers 57: Quantitative Applications in the Social Sciences. Beverly
Hills, CA: Sage.
Scriven, M. (1991). Evaluation Thesaurus, 4 th Ed. Newbury Park, CA: Sage.
Stecher, B. M. & Davis, W. A. (1987). How to Focus an Evaluation. Newbury Park, CA: Sage.
Torres, R. T., Preskill, H. S. & Piontek, M. E. (1996). Evaluation Strategies for Communicating and
Reporting: Enhancing Learning in Organizations. Newbury Park, CA: Sage.

Wadsworth, E. M. (Ed). (1996). Evaluation Resource Book. West Lafayette, IN: Women in Engineering
Program Advocates Network.
Weiss, C. H. (1997). Evaluation. Englewood Cliffs, NJ: Prentice Hall.
Wholey, J. S., Newcomer, K. E. & Hatry, H. P. (2007). Handbook of Practical Program Evaluation. San
Francisco, CA: Jossey-Bass.
Worthen, B. R. & Sanders, J. R. (1987). Educational Evaluation: Alternative Approaches and Practical
Guidelines. White Plains, NY: Longman.