Citizen Report Cards to Improve Service Delivery - South Africa ...

South Africa
Citizen Report Cards to Improve Service Delivery
A Preliminary Outline
Monitoring and Impact Evaluation Conference
The Presidency and The World Bank
June 19-23, 2006
Background
Too often, services fail poor people—in access, in quantity, in quality. But the fact that there
are strong examples where services do work means governments and citizens can do better.
How? By putting poor people at the center of service provision: by enabling them to monitor
and discipline service providers, by amplifying their voice in policymaking, and by
strengthening the incentives for providers to serve their clients.
The public sector has generally taken on responsibility for the delivery of public
services and frequently used civil service bureaucracies as the instrument. This approach has
had dramatic successes and many failures across the world. Particularly for poor people,
there are widespread challenges in providing affordable access, fixing dysfunctional facilities,
improving technical quality, increasing client responsiveness, and raising productivity.
Neither economic growth, nor simply increasing public spending, nor coming up with
technocratic solutions is enough to meet this challenge.
The failures in service provision have not gone unnoticed. Indeed, there is a whole
host of proposed institutional solutions: civil service reform, privatization, democratization,
decentralization, contracting out, provision through NGOs, empowerment, participatory
methods, social funds, community driven development, user associations. With each of these
solutions comes a variety of techniques and instruments: demand-side transfers, participatory
rural appraisals, facility surveys, participatory budgets. None is a panacea.
To analyze service delivery, it is helpful to have a simple conceptual framework. Let
us start with the specific, such as a child in a classroom, a pregnant woman at a clinic,
someone turning a tap for water. Each is seeking a service, and the proximate determinants of
success are clear. For any individual service transaction to be successful, there needs to be a
frontline provider who is capable, who has access to adequate resources and inputs, and who
is motivated to pursue an achievable goal. The general question: what institutional
conditions support the emergence of capable, motivated frontline providers with clear
objectives and adequate resources? The answer: successful services emerge from institutional
relationships in which the actors are accountable to each other.
The Relationships of Accountability
For the conceptual framework, we distinguish four relationships of accountably in service
delivery: those between citizen/clients, politicians, provider organizations, and frontline
service providers.
1

Of politicians to citizens. The term voice is used to express the complex relationships
of accountability between citizens and politicians. Voice is about politics, but it covers much
more. The voice relationship includes formal political mechanisms (political parties and
elections) and informal ones (advocacy groups and public information campaigns).
Delegation and finance between citizen and state are the decisions about pursuing collective
objectives and mobilizing of public resources to meet those objectives. Citizens need
information about how actions of the state have promoted their well-being. They also need
some mechanism for enforceability, to make sure that politicians and policymakers are
rewarded for good actions and penalized for bad ones. If politicians have abused their
position, or even just not pursued objectives aggressively and effectively, citizens need a
variety of mechanisms—not just periodic elections—to make politicians and policymakers
accountable.
Of the organizational provider to the state. The relationships between policymakers
and service providers can be thought of as compacts. The compact is not always as specific
and legally enforceable as a contract, though a contract can be one form of a compact.
Instead, it is a broad agreement about a long-term relationship. The policymaker provides
resources and delegates powers and responsibility for collective objectives to the service
providers. The policymaker generates information about the performance of organizations.
Enforceability comes into play when the compact also specifies the rewards (and possibly the
penalties) that depend on the service provider’s actions and outputs. The line between “the
state” and “public sector organizational provider” is not always easy to draw.
Of the frontline professionals to the organizational provider and its management. In
every organization formal and informal tools of management provide frontline workers with
assignments and delineated areas of responsibility, equipping them with the resources to act.
In public agencies this management function is at times blurred because providers are
employees of “the government.” But all the standard management issues of selecting,
training, and motivating workers in an organization apply to all organizations—private, NGO,
government, whatever. All service provision organizations—whether a government ministry,
a religious body, a nonprofit NGO, or a for-profit firm—have to create a relationship of
accountability with their frontline providers.
Of the provider to the citizen-client: client power. Because the policymaker cannot
specify all actions of providers in the compact, citizens must reveal to providers their demand
for services and monitor the providers’ provision of services. Clients and organizational
providers interact through the individuals who provide services—teachers, doctors, engineers,
repairmen—the frontline professionals and frontline workers.
How is accountability achieved in the public sector? To begin to answer this question,
compare how accountability is achieved in many market contexts. In the market, dissatisfied
consumers can successfully use the exit option; i.e., if the price is too high or the quality too
low, the consumer can choose not to buy the good or buy from another producer. If many
consumers act in the same way, this will influence the producer’s profitability and, in the end,
its survival in the market. The exit mechanism, however, may not work well in the public
sector. First of all, in some cases there may be no easily available alternative to the local
public provider. Second, the link between the public provider’s performance and its financial
position (or its staff’s remuneration) is typically weak or non-existent.
2

Governments have tried to compensate the lack of a well-functioning exit mechanism
by increasing control. That is, create or strengthen legal and financial management methods
to monitor public sector performance. This is a top down approach where some government
agencies are assigned to control and monitor others. Evidence from a wide spectrum of
countries, however, shows that these systems can, at best, provide only a limited form of
monitoring.
A complementary approach emphasizes citizens’ voice and “client power” as
important elements of making services work for poor people. When the exit option has no
bearing or is weak, strengthening the voice mechanism may be the only available option. In
recent years, civil society organizations and others have experimented with various
interventions this general objective in mind. A common thread across these interventions is
that they take the users of public services as a starting point. Thus, rather than focusing on
service providers' accountability to policymakers alone, the idea is to engage citizens at the
bottom of the public service delivery chain by providing easy access to information on the
workings of public programs intended for their benefit. This empowers citizens to demand
certain standards, to monitor service quality, and to challenge abuses by officials with whom
they interact.
The topic of this project proposal, the Citizen Report Card, is one of the new tools
aimed at increasing client power over frontline providers and hence at improving services.
Citizen Report Cards: What Are They and Why Use Them?
Some information strategies look directly at public service outputs (quality and quantity of
services provided by government) rather than inputs (prices paid, budgets committed and
disbursed). The best known are the citizen report cards developed by the Public Affairs
Centre in Bangalore, India. 1 Citizens are asked to rate service access and quality and to report
on concerns about public services, general grievances, and corruption. Citizen report cards
have spread beyond Bangalore to cities in Kenya, Mozambique, the Philippines, Ukraine, and
Vietnam. They have been scaled up in India to cover urban and rural services in 24 states. In
Uganda, they have been applied to health care at the local level in rural areas, and are
currently being evaluated rigorously for impact. Overall, the citizen report cards have
stimulated considerable media, bureaucratic, and political attention and there is general
acknowledgment of their positive contribution to service improvements.
Because citizen report cards focus on service outcomes, they do not necessarily
provide citizens/clients with information about specific decisions that policymakers have
taken—or not taken. Nor do they give them information (at least in their first round) about
service benchmarks, except to the extent that the agencies themselves have established service
standards (but repeat report cards do provide implicit benchmarks from the previous report
card). So it can be hard for citizens/clients to assess, on the basis of one report card, whether
the results justify voting against the incumbents at the next election.
Citizen Report Cards in Bangalore seem to have had a more direct influence on the
heads and senior managers of the municipal and utility agencies responsible for services. The
high visibility of report cards in the press and civic forums turned them into league tables of
1 Samuel Paul, 2002, Holding the State to Account: Citizen Monitoring in Action. Bangalore: Books for Change.
3

the efficacy of municipal agencies. The reputational competition arising from the report cards
is enhanced by joint agency meetings attended by prominent social and political leaders and
citizens.
A Report Card is a tool to collect feedback from the users (and potential users) of
public services and disseminate this information back to the citizens/users so they have
reliable information about how their neighborhood/community at large views the quality and
efficacy of service delivery, and so they can compare service delivery in their neighborhood
vis-à-vis other neighborhoods in their city, larger metropolitan area, or across cities or
municipalities in the country at large. The basic idea is that reliable and representative
information about the users’ experiences and entitlements is critical for citizens’ ability to
monitor service providers. It also improves users’ ability to challenge abuses of the system,
since reliable quantitative information is more difficult for service providers to brush aside as
anecdotal, partial, or simply irrelevant.
Practically, a Report Card has three components: (a) collecting quantitative
information from users (citizens) and service providers, using micro survey techniques; (b)
assembling this information in “easy access/comprehensible report cards”; (c) disseminating
the report cards to users and providers and providing them with practical information on how
best to use this information; and (d) implementing repeat user and provider surveys to assess
impact on service delivery outcomes.
Proposal for Consultative Report Card in Municipal Services in South Africa
Currently, a Consultative Report Card initiative is being trialed in South Africa in the metro
Tshwane area (questionnaire attached), in collaboration with the National Treasury and the
Human Sciences Research Council (HSRC). Following the pilot, the plan is to extend the
Report Cards to another 40 or so municipalities across South Africa, rural and urban. In all
instances, the Report Cards will gather citizens’ input on the core services which
municipalities are mandated to deliver, such as water, sanitation, waste disposal, etc. A
critical element of the Report Card process will be to ensure there is effective dissemination
of the data and information gathered from the Report Card to citizens of that municipality –
they, after all, will be at the crux of demanding and pushing for better service delivery from
the municipality. To assess the efficacy of the Report Card in stimulating citizens’ voice and
pressure for change, another 40 municipalities will be selected as control group. This
approach will allow a rigorous impact assessment to be undertaken to determine whether
citizens’ access to information is, indeed, an important factor responsible for positive changes
recorded in service delivery. Ideally, the Report Card activity will result in citizens in those
municipalities having solid and reliable information about the coverage, quality, and
efficiency of services in their own neighborhood/municipality and the information necessary
to assess service delivery in their locality relative to the other municipalities and to the
national average. Advocacy based on such information can then lead to real improvements on
the ground.
The Tshwane pilot Report Card pilot is expected to be completed by the end of
October 2006. The expansion to other municipalities will begin as of November 2006,
including surveys of users and service providers. Dissemination activities will take place in
mid-2007 (and will possibly be repeated after a few months). The repeat surveys of users and
providers to assess the impact of this intervention will be implemented in June-July 2008.
4

Impact Evaluation of the Report Cards
Identifying and implementing incentives that give rise to a strong relationship of
accountability between service providers and beneficiaries is viewed by many as critical for
improving service delivery. How to achieve this, however, is less understood. Systematic
evaluation of service delivery innovations to increase accountability can show what works,
what doesn’t and why, a first step to scaling up successes.
There are many different ways to evaluate a program/project, each subject to more or
less serious constraints. An impact evaluation attempts to answer the counterfactual question:
how would individuals or municipalities that participated in a program, for instance, the
Report Cards, have fared in the absence of the program? The difficulty with this question is
immediate: at a given point in time, an individual or municipality is observed to either be
exposed or not exposed to the program. It is therefore not possible to obtain an estimate of the
impact of the program on each individual or municipality. However, it may be possible to
obtain the average impact of the program by comparing participating municipalities (the
treatment group) with a similar group of municipalities who are not exposed to the program
(the control group). One implication of this is that an evaluation based on statistical methods
requires are “large” sample. This is one reason why the Report Card project will take its
starting point the municipal level.
The problem facing an evaluator is how to design the evaluation study so as to
establish a credible comparison group -- a group of municipalities who in the absence of the
program would have had outcomes similar to those who were exposed to the program. This
group should give us an idea of what would have happened to the members of the program
group, if they had not been exposed to the Report Card, and thus allow us to obtain an
estimate of the average impact on the group in question.
To solve this problem, program evaluations typically need to be carefully planned in
advance in order to determine which group is a likely control group. One credible way to
ensure that the treatment and comparison groups are similar is to select randomly from a
potential population of participants (such as municipalities). If the sample is large enough, one
can then be assured that, on average, those who are exposed to the program are no different
than those who are not, and thus that a statistically significant difference between the groups
in the outcomes the program was planning to affect can be confidently attributed to the
program.
To evaluate a program, one also needs to have a quantifiable outcome measure(s).
This, in turn, requires knowledge of how the program possibly could influence outcomes. In
this particular case, we require knowledge about how households with access to information
on how the local providers, and/or local government, perform (and what citizens are entitled
to) can possibly influence the services they receive. A necessary condition for this to be the
case is that the local providers and/or local governments must be able to influence the
outcomes that are measured and that one of the reasons the initial outcomes are suboptimal is
related to choices made by the provider or local government (for example, budgets are spent
on items not benefiting the intended beneficiary, or low effort is exerted in executing local
programs).
5

The background study on the supply side of service delivery, with a focus on the local
government level, was commissioned to assist in identifying what type of services possibly
could be included in the Report Card project. 2 Similarly, focus groups of users, frontline
service providers, and local politicians are being carried out in May-June 2006 in eight
randomly selected municipalities to get a better sense of issues faced on the ground by both
users and providers.
Extension of Impact Evaluation to the “Compact”
More accurate information on performance and entitlements may not be enough to strengthen
accountability in the public sector. In fact, transparency can just produce cynicism if those
misbehaving operate with impunity. Thus, information provision will not work unless there
are members of the public willing to make use of the information and if citizens can either
directly or indirectly sanction the officials or their superiors.
One way to test this hypothesis would be to not only introduce variation in what type
of information is disseminated back to households, but also experiment with increased
monitoring from upper-level government authorities. In this case, one must identify services
that are provided by the municipality (or local providers) but where provincial or national
agencies also have formal role in monitoring (and possible sanction) the local providers.
RReinikka/JSvensson/CWinter
18/06/2006
2 Doreen Atkinson, 2006, “The Status of Service Delivery and Impacts in South Africa:
A Preliminary Overview” World Bank. Processed.
6