OTIS Evaluation Guide (PDF) - California Department of Public Health

D. How To Find A Local Program Evaluator
You may find many ways to land a qualified evaluator for your program. The easiest thing to do, of course, is to
re-sign the evaluator you worked with in the previous funding cycle, or to appoint a staff member in your program.
However, if you want to identify a new evaluator, a little homework may be necessary. Following are some
tips to help you locate your dream evaluator.
1. Using the Local Program Evaluator Directory
TCS has an on-line Local Program Evaluator Directory (http://icswebdev.esp.fsu.edu/development/LPEDirectory/site/).
Interested evaluators can set an account in the directory and enter their information. A “search”
function that enables local program staff to search the entire directory with any keywords. For example, if a
project director wants to find an evaluator who has experience with Hispanic/Latino populations as well as
secondhand smoke related projects, she can type in “Hispanic” or “Latino” and “secondhand smoke” in the
textbox for searching. There will be a great chance that the names of evaluators who meet both the criteria
will be shown on the results page.
The directory is a powerful tool to quickly locate and get background information of evaluators with specific
requirements. However, the directory only provides one-dimensional information about your future evaluator.
How good is he or she in a real project?
2. Communicating with peers
Word of mouth communication provides you with some insights that are not retrievable from the directory. It
is very common that program directors recommend evaluators to peers based on a satisfactory experience.
They will tell you how easy it is to work with a particular evaluator; how reliable the evaluator is; how flexible
the evaluator is; and so on. Such information adds another dimension to your decision-making process. However,
experience and opinion from other people cannot predict what your experience will be.
3. Optimizing the “First Contact”
Before making any decision, it will be in your best interest to have a brief interview with your evaluator
candidates. Just like a job interview, you will not only get a sense of the quality of the candidates, but also
build some rapport with the potential partner. How do you find out whether the candidate is qualified or not,
besides a general impression? One tool we recommend is the Local Program Evaluation Planning Guide (the
Guide). If, as a project director, you have read the first half of the Guide and familiarize yourself with some
evaluation “lingo”, you might be able to ask some interesting questions related to your efforts.
After these three steps, you should feel confident that you have a 3-D view of your future evaluator.
No matter how you select the evaluator, it is recommended that the evaluator should have the following qualifications:
Recommended minimum requirements:
• One course in study design or one year of experience determining the study design for an evaluation.
The skills needed include sample size calculations, sampling scheme and data collection methods.
• One course in evaluation or one year of experience planning and implementing an evaluation. The skills
needed include knowledge of process, impact, and outcome evaluation.
• Two courses in statistics or one year of experience analyzing data for an evaluation. The skills needed
include descriptive and inferential statistics, use of computer software for database management, and use
of computer packages for analysis.
138
OTIS Evaluation Guide • March 2007