Citizen Report Cards to Improve Service Delivery - South Africa ...

Impact Evaluation of the Report Cards
Identifying and implementing incentives that give rise to a strong relationship of
accountability between service providers and beneficiaries is viewed by many as critical for
improving service delivery. How to achieve this, however, is less understood. Systematic
evaluation of service delivery innovations to increase accountability can show what works,
what doesn’t and why, a first step to scaling up successes.
There are many different ways to evaluate a program/project, each subject to more or
less serious constraints. An impact evaluation attempts to answer the counterfactual question:
how would individuals or municipalities that participated in a program, for instance, the
Report Cards, have fared in the absence of the program? The difficulty with this question is
immediate: at a given point in time, an individual or municipality is observed to either be
exposed or not exposed to the program. It is therefore not possible to obtain an estimate of the
impact of the program on each individual or municipality. However, it may be possible to
obtain the average impact of the program by comparing participating municipalities (the
treatment group) with a similar group of municipalities who are not exposed to the program
(the control group). One implication of this is that an evaluation based on statistical methods
requires are “large” sample. This is one reason why the Report Card project will take its
starting point the municipal level.
The problem facing an evaluator is how to design the evaluation study so as to
establish a credible comparison group -- a group of municipalities who in the absence of the
program would have had outcomes similar to those who were exposed to the program. This
group should give us an idea of what would have happened to the members of the program
group, if they had not been exposed to the Report Card, and thus allow us to obtain an
estimate of the average impact on the group in question.
To solve this problem, program evaluations typically need to be carefully planned in
advance in order to determine which group is a likely control group. One credible way to
ensure that the treatment and comparison groups are similar is to select randomly from a
potential population of participants (such as municipalities). If the sample is large enough, one
can then be assured that, on average, those who are exposed to the program are no different
than those who are not, and thus that a statistically significant difference between the groups
in the outcomes the program was planning to affect can be confidently attributed to the
program.
To evaluate a program, one also needs to have a quantifiable outcome measure(s).
This, in turn, requires knowledge of how the program possibly could influence outcomes. In
this particular case, we require knowledge about how households with access to information
on how the local providers, and/or local government, perform (and what citizens are entitled
to) can possibly influence the services they receive. A necessary condition for this to be the
case is that the local providers and/or local governments must be able to influence the
outcomes that are measured and that one of the reasons the initial outcomes are suboptimal is
related to choices made by the provider or local government (for example, budgets are spent
on items not benefiting the intended beneficiary, or low effort is exerted in executing local
programs).
5