Dr. Eva Terberger - OneWorldConvention

Evaluation of Development
Impact:
Measurability and Efficiency of
Microfinance Development Results
Prof. Dr. Eva Terberger
Universität Mannheim
21st of May 2011
One World Convention on Social Business
Passau

The Microfinance-Promise
„About one billion people globally live in households with per capita income
of under one dollar per day. The policymakers and practitioners who have
been trying to improve the lives of that billion face an uphill battle...
Amid the dispiriting news, excitement is building about a set of unusual
financial institutions prospering in distant corners of the world - especially
Bolivia, Bangladesh, and Indonesia....These institutions - united under the
banner of microfinance - share a commitment to serving clients that have
been excluded from the formal banking sector.“
Morduch, J. (1999): The Microfinance Promise, JEL, pp. 1569-1614, p.
1569

Microloans as directed preferential credit (subsidized
interest rates)
+ to fight poverty
+ to foster economic development and growth
Ambitions of
microfinance in
developing
countries in the
1970s
Dismal results
New approach
Market oriented Microfinance – subsidies for building
microfinance institutions (institution building)
+ as a contribution to financial market development
+ as a „low cost“ and in the longer run „cost
covering“ development strategy (sustainability)
„Microfinance appears to offer a „win-win“
solution, where both financial institutions
and poor clientele profit“ (Morduch 1999, p. 1570)
Additional
ambitions of
microfinance in
developing and
transformation
countries since the
end of the 1980s

The Role of Building Microfinance-Institutions: Developing the
Financial Market
Formal Financial Sector
Micro/Small
Borrowers
• More investment
• Higher income
• job creation
Microfinance-Institution
(cost covering or even profitable)
Informal Financial Sector
(money lenders, suppliers, land owners)
Poverty Reduction and Growth
Low income
savers
• Improved risk
management
• Savings
mobilization

The Microfinance Promise – An illusion
• Headlines about MFIs going public: Compartamos, Mexico
2007 und SKS, India 2010.
Mission Drift (Is microfinance a „Business like WalMart“)
• Financial Crisis: High PaR and NPL of MFIs in Nicaragua,
Bosnia, Marocco…
Is microfinance driving the poor into over-indebtedness
Microfinance as a Ponzi scheme
• Rigorous (statistical/econometrical) impact studies: Impact of
microfinance is smaller than microfinance supporters have
claimed in the past.
Microfinance: Is the „Emperor of Poverty Reduction“
actually naked

Overview
I. The Microfinance Approach – An Emperor Without Clothes
II.
Evaluation of Development Impact
o
o
What for
How
III. Microfinance: From Inputs via Outputs to Outcomes and
Impacts
o
o
The Impact Chain
Measurability of Microfinance Development Impact
IV. The Efficiency of Microfinance Interventions: What about
Cost-Benefit-Ratios
V. Some Evaluation Examples

Evaluation:
Definition and Objectives
Definition
An assessment, as systematic and objective as possible of an ongoing
or completed project, programme or policy, its design,
implementation and results. The aim is to determine the relevance
and fulfilment of objectives, developmental efficiency,
effectiveness, impact and sustainability. An evaluation should
provide information that is credible and useful, enabling the
incorporation of lessons learned into the decision-making process
of both recipients and donors. (DAC 1991)
Objectives
1. Accountability (not in the sense of auditing):
To which extent do the projects meet their objectives (development impact)
2. Learning how to do better:
What lessons can be drawn from (positive and negative) experiences

The question on development impact is difficult to answer!
Why
What we would like to know:
How big were (ex ante: will be) the development results (outcome/impact) per input unit
Where and how will we achieve the highest development impact per unit
Challenges
• How do we know which changes can be attributed to an intervention
• How can we distinguish whether the result of an intervention is a systematic
one or produced by the particular circumstances
• What about unexpected results
• Where do we get the standard from what development results are or will be
satisfactory (comparison between alternatives)
Evaluation is a continual process of learning how to measure and assess
development results!

Impact chains and the evaluation criteria of the Development
Assistance Committee (DAC) of the OECD
Impact Chains
Input
Money, Human Resources
Output
Classrooms, water supply system...
DAC – Evaluation-Criteria:
„Relevance“
Is intervention suited to mitigate
a development impediment
„Effectiveness“
Were the intervention‘s objectives
achieved
Outcome
Higher enrolment rates, use of safe water...
Impact
Higher incomes, better health, less poverty...
„Impact“–
Was the intended improvement
of living conditions achieved
„Efficiency“
Were the benefits satisfactory
in relation to inputs
„Sustainability“
Are benefits expected
to keep on flowing
Source: KfW Entwicklungsbank, Evaluation
Department FZ E
9

The DAC-Criteria are not the solution to all evaluation challenges
Impact Chains
Input
Money, Human Resources
DAC – Evaluation-Criteria:
„Relevance“
Is intervention suited to mitigate
a development impediment
If it was possible to answer the
question on efficiency, one
would not need the other
criteria.....
Output
Classrooms, water supply system...
Outcome
Higher enrolment rates, use of safe water...
„Effectiveness“
Were the intervention‘s objectives
achieved
„Impact“–
Was the intended improvement
of living conditions achieved
Impact
Higher incomes, better health, less poverty...
„Efficiency“
Were the benefits satisfactory
in relation to inputs
„Sustainability“
Are benefits expected
to keep on flowing
Source: KfW Entwicklungsbank, Evaluation
Department FZ E
10

• The DAC-Criteria are a suited framework to structure evaluation
processes and evaluation findings, but no more.
• Evaluations have to rely on a multitude of methods, theories,
sources...
Evaluators are like detectives looking for different hints and puzzle
pieces...
...to tell a plausible „story“ based on evidence.
Or has this changed since new methods of impact measurement are
applied

Measuring net impacts via control group
Comparison before vs. after:
Determination of net impact:
Impact Indicator
TG
after
intervention
before
intervention
Impact Indicator
Dt1
Dt2
TG
net
Impacts
CG
t 1 t 2
TG: Target Group, t: time
time
t 1 t 2
time
TG: Target Group, t: time
CG: Control Group, D: Difference TG – CG
Source: KfW Entwicklungsbank, Evaluation
Department FZ E

For rigorous impact measurement you need
• A control group
• Lots of data (target group and control group)
• A well defined intervention
• ....
And even if you manage to do a rigorous impact measurement, you do
not know...
• whether the results are transferable to other settings,
circumstances...(systematic reviews try to tackle that problem by
giving an overview about all rigorous impact assessments of
similar interventions)
• whether the interventions have got a good cost-benefit-ratio
(efficiency!!)

Overview
I. The Microfinance Approach – An Emperor Without Clothes
II.
Evaluation of Development Impact
o
o
What for
How
III. Microfinance: From Inputs via Outputs to Outcomes
and Impacts
o
o
The Impact Chain
Measurability of Microfinance Development Impact
IV. The Efficiency of Microfinance Interventions: What about
Cost-Benefit-Ratios
V. Some Evaluation Examples

The Impact Chain of Microfinance Projects
Institution
Project Measures (Input):
Loan, funds, fiduciary/equity holding, capacity development
Project Results (Output):
Banks grant loans to MSMEs with adequate credit technology
Financial System
and Real Economy
Project Objective (Outcome):
Population segments formerly unserved by the formal financial sector,
microentrepreneurs in particular, have access to financial services
Micro-level: Financial products for target group
Productive investments in MSMEs
Children go to school
Consumption smoothing
Empowerment
Overall Objective (Impact):
Macro-level: Financial sector development
Demonstration effects
Achieving MDGs, improving living conditions, economic growth
Source: KfW Entwicklungsbank, Evaluation Department FZ E
Higher demand for labor, goods, and
education
Higher education levels
Better legal and governance system

Impact chain and Ex Post-Evaluations:
Methodological Assessment
Institution
Measures (Input):
Loan, funds, fiduciary/equity holding, capacity
development
Results (Output):
Banks grant loans to MSMEs with adequate credit
technology
Project Objective (Outcome):
Population segments formerly unserved by the formal financial
sector, microentrepreneurs in particular, have access to financial
services
Methods applied
• Evaluation of credit technology using
financial ratios (e.g. repayment rates)
and by analyzing internal processes
• Evaluation of governance structure
and assessment of shareholders
• Analysis of target group focus through
loan amounts, financial products and
location of branches
• Analysis of banking sector, market
structure, banking regulation and
financial sector indicators
► Analysis with simple quantitative
and qualitative methods
► High validity of results especially if
something does not work
► Useful results for policy makers!
Source: KfW Entwicklungsbank, Evaluation
Department FZ E

Impact chain and Ex Post-Evaluations:
Methodological Assessment
Financial System
and Real economy
Project Objective (Outcome):
Micro -level: Financial products for target group Macro -level: Financial sector development
Productive investments in
MSMEs
Demonstration effects
Children go to school
Higher demand for labor, goods, and
education
Consumption smoothing
Empowerment
Higher education levels
Better legal system
Overall Objective (Impact):
Achieving MDGs , improving living conditions, economic growth
Methodological Assessment
Problem Micro-level
‣“Self-selection“: Personal differences increasing the
likelihood that someone receives a loan (e.g.
education, entrepreneurial ability) may also
influence what the impact assessment is looking for,
e.g. income.
‣ Able entrepreneur would have had higher income
even without credit
Results of net impact measurement
Problem Macro-level
‣„Endogeneity“: MFIs are usually not selecting
regions or villages randomly, but based on
characteristics related to program success.
‣ „Unobserved Heterogeneity“: Cross-country
studies can not control for all differences between
countries (financial sector development only one
factor).
Standard Ex Post-Evaluations (rapid appraisal) can only use proxies!
Source: KfW Entwicklungsbank, Evaluation Department FZ E

Robust Impact Assessments: Method and Results
• To measure net impacts, we always need a control group:
‣ Experimental designs (randomized control trials)
‣ Loans are only granted to a randomly selected group of clients (or
regions), whereas the control group (control region) has no access to
finance.
‣ Quasi-experimental designs
‣ Compare client and control groups with complex econometric
techniques (e.g. difference-in-difference, matching, instrumental
variables, regression discontinuity design etc.)
‣ Validity of results can be close to experimental designs

• Up to now, there exist less than a handful of rigorous studies on impact of
microfinance interventions on livelihoods (no systematic review was
possible up to now, because of insufficient number of studies)
• The studies which are there measure short term impact (12 to 18 months),
not long term effects on livelihoods
• What these studies show:
• Study in India by Banarjee, Duflo et al. 2009 : Access to microloans
lead to more new businesses, more durable assets, higher profits...no
higher income, more education. (Introduction of formal microfinance,
but moneylenders were there)
• Study in the Philippines by Karlan and Zinman 2009: Access to
microloans replaced informal finance, no more investments, but more
education (treatment effect bigger for males with higher income)
• Study on savings in Kenia 2008 by Dupas and Robinson: Increases in
investment and expenditure for women who use accounts.
• Study by Duflo et al in Marocco coming up: Similar results....

Will there be more studies in the future
Certainly yes. However...
• These studies have difficulties to trace longer term effects
• It is becoming more and more difficult to find control groups
• Effects on employment, governance, etc are difficult to measure…
• It needs to be kept in mind, however, that loans go along with a risk for
the borrower of not being able to pay (over-indebtedness).
• Principles of responsible finance followed by MFI and financial literacy
for clients help to mitigate risk of clients‘ over-indebtedness!

The macroeconomic perspective
• Growth as the driving force of poverty reduction
• Financial market development as a driving force of
growth (World Bank 2002, Strategy Research Report: Finance for
Growth)
• Better institutions as a driving force of growth
(Acemoglu 2003)
• More equity in access and opportunities as a driving
force of growth (World Bank, Development Report 2006: Equity
and Development)
• Financial market development goes along with more
equity of income (Beck et al. 2007)

Beck, Demirgüc-Kunt, Levine (2007): Finance, Inequality and the
Poor
Abstract: Financial development disproportionately boosts incomes of
the poorest quintile and reduces income inequality. About 40% of the
long-run impact of financial development on the income growth of the
poorest quintile is the result of reductions in income inequality, while
60% is due to the impact of financial development on aggregate
economic growth. Furthermore, financial development is associated with
a drop in the fraction of the population living on less than $1 a day, a
result which holds when conditioning on average growth. These findings
emphasize the importance of the financial system for the poor.
Microfinance –
A tool to induce Pro-Poor-Growth

Overview
I. The Microfinance Approach – An Emperor Without Clothes
II.
Evaluation of Development Impact
o
o
What for
How
III. Microfinance: From Inputs via Outputs to Outcomes and
Impacts
o
o
The Impact Chain
Measurability of Microfinance Development Impact
IV. The Efficiency of Microfinance Interventions: What
about Cost-Benefit-Ratios
V. Some Evaluation Examples

Even if positive outcomes and impacts are small…
…they flow for many, many years, if MFIs sustainably deliver financial
services
Just remember: The German savings
banks and credit unions were the
microfinance institutions of yesterday!
And how much does it cost (inputs) to initiate the
flow of benefits

Mikrofinance -
Development stages
1980
1995
2009
Starting with grant
financed funds,
which often lost money
...moving on to MFIs,
achieving cost coverage
Microcreditportfolio
Microcredit-
...moving on to investment vehicles,
promising moderate profits...
Microcreditportfolio
portfolio Microcreditportfolio
Grants
Microfinanceinstitution
Microfinanceinstitution
Micro-
financeinstitution
MFI
MFI
MFI
Microcreditportfolio
Mikro- Microkredit-
credit-
portfolio Mikrokredit-
Microportfolio
creditportfolio
MFI
MFI
MFI
Grants and
socially
motivated
investors
MIV
MIV
MIV
Grants and
socially
motivated
investors

STRUCTURED FINANCE
• Complementary roles for public and private grants and for
social investment (and even comercial investment)
• Producing a high „benefit leverage“ with scarce ressources
Microcreditportfolio
MFI
MFI
MFI
MFI
MFI
MFI
MFI
MFI
Investmentfunds/Asset Backed Securities
Portfolio of
Microfinance
-investments
First Loss Piece
(Equity)
Mezzanine
Tranche
Senior Tranche
Risk-Cascade
Grants
(or even comercial
risk capital)
Development Banks
Private Social
Investors
(and comercial Investors)

If microfinance is accompanied by the promotion of responsible finance
and embedded into an approach of building sound financial systems
(deposit insurance, credit bureau, banking supervision)
… the positive impact is very likely to outweigh potential negative effects
(like the risk of overindebtedness) and, as a consequence,
…the microfinance approach will have a high cost-benefit-ratio (efficiency)
in its favour!!!

Overview
I. The Microfinance Approach – An Emperor Without Clothes
II.
Evaluation of Development Impact
o
o
What for
How
III. Microfinance: From Inputs via Outputs to Outcomes and
Impacts
o
o
The Impact Chain
Measurability of Microfinance Development Impact
IV. The Efficiency of Microfinance Interventions: What about
Cost-Benefit-Ratios
V. Some evaluation examples

Beispiele Ex Post Evaluation at KfW – Some Examples
Mauretania: Development Bank UNCACEM
(agricultural cooperative bank, only bank for
agricultural finance)
• Credit line 1,28 Mio. EUR as refinance for
short and medium term agricultural loans for
working capital and asset finance,
• Credit line was reduced to 0,88 Mio EUR
because condition to collateralize loans was
not met.
• Loan recipients belonged to the target group
(average loan amount 13.500 EUR)
• Very high delinquency rates: Portfolio at risk of
46% (>90T), the rest of the performing
portfolio contains many restructured loans
• Real interest rates were negativ; „cost
coverage“ only possible due to subsidies (soft
loans)
Source: KfW Entwicklungsbank, Evaluation
Department FZ E
Ex Post Evaluation 2007
Rated as overall unsatisfactory,
with some, however too little,
positive effects

Beispiele Ex Post Evaluation at KfW – Some Examples
Azerbaidjan: Access Bank AZE
• Fiduciary equity 2,65 Mio. EUR and 1,7 Mio
EUR TA (for institution building)
• Bank founded in 2002 as shareholding
company, shareholders KfW (for the German
Government), IFC, EBRD, BSTDB as well as
Access Holding und Management Consultant
LFS
• Bank offers the full range of financial services
an (very successful in rural and agricultural
finance), Portfolio at Risk < 1%.
• Bank is the only bank with a full banking
licence to focus on small and micro enterprise.
• Bank is covering cost and makes a moderate
profit.
Ex Post Evaluation 2009
Gesamtbeurteilung: very successful
In August 2010 Euromoney awarded
AccessBank as the best bank of Azerbaidjan
Source: KfW Entwicklungsbank, Evaluation Department FZ E

Thank you!