Inside the Black Box - Chapin Hall at the University of Chicago

The organizations also reported that they use staff evaluations and participant surveys to assess and
improve program performance to better serve their participants.
The Need for Better Program Measures
All of the programs who participated in this study emphasized the need to track more comprehensive and
meaningful outcome measures that better reflect participant accomplishments, progress, and program
impacts. Specifically, many interviewees were interested in developing a more multidimensional picture
of their participants. They wanted a better idea of their participants’ individual characteristics, life
circumstances, and barriers to success; a deeper understanding of specific attributes of their program
model; and they wanted to incorporate, and even exceed, the data requirements of their federal and private
funders.
Several program interviewees wanted to measure the outcomes of their participants over a longer period
of time than is currently measured in order to see longer-term changes in employment outcomes.
One of the biggest mistakes of foundations in general, even the public sector, is that…they want
immediate outcomes and we can produce immediate outcomes, but not with effectiveness that will
change the lives of the participant.
Program staff also noted that current measures do not account for programs that work with harder-toserve
populations that have barriers to employment. They expressed a desire to provide specific measures
relevant to the populations they are serving and for performance expectations to be adjusted accordingly.
Another point raised in the interviews concerned tracking participants who do not complete the program.
Although funders are primarily interested in outcomes of those who complete the program and place into
employment, one interviewee noted the potential value of tracking outcomes of those individuals who
may have exited the program early or exited without a job placement. One interviewee expressed interest
in measuring participant resiliency and assessing participants who may drop out due to challenging
circumstances, but later reengage in the program.
Furthermore, in regard to measures required by funders, multiple organizations noted the difficulty of
attributing an individual’s outcomes to a specific funding source, when funding from multiple sources is
often used to support the individual as they progress through the program. Interviewees from multiple
organizations said they are also looking for ways to tie outcomes to specific services offered, but not all
support services are formally tracked. One of the providers believed that there was a positive impact from
case management, but acknowledged the challenge in measuring and proving that impact.
Providers in the study also noted that they were looking for ways to measure participant outcomes beyond
job placement, retention, and wages, since data about these areas provide only a narrow view of program
Chapin Hall at the University of Chicago 19