Microeconometric evaluation of active labour market policies - SFI

Microeconometric evaluation of
active labour market policies
Nordic Conference on Effects of Education and Labour Market Policies,
Copenhagen, 2004
Michael Lechner
SIAW, CEPR, IZA, ZEW
www.siaw.unisg.ch/lechner

Structure of talk
Part: 0: The identification problem
Part I: 4 types of papers on the effects of ALMP in CH
- different methods
- different data
- different questions
Part II: Overview of econometric estimators used for evaluation

Introduction to selection bias and causal effects
Many microeconometric evaluation studies of active labour market policies in
Europe (e.g. ‘old’ survey by Heckman, LaLonde, Smith, 1998, Handbook of
Labour Economics, 4)
Problems of all studies
– Case workers select specific types of unemployed into specific programmes
– Specific unemployed self-select into specific programmes
– Programmes may have effect on labour market outcomes
Labour market outcomes are correlated with participation, but why?
How can we disentangle selection and programme effects?

The identification problem
– Nonparticipants may not be comparable to participants
– Participants before participation may not be comparable to participants after
participation
– These problems might persist even after conditioning on observable attributes
Bad news: required assumptions are untestable!
Even worse: different identifying strategies identify different effects!
Important first steps in every evaluation study:
- What do we want to estimate?
- How can we translate desired effect into a statistical causal parameter?
- What do we need to know to be able to estimate this parameter?

Three solutions to the identification problem with
heterogeneous effects
– Postulate parametric model for the dependence of the outcome variable on
selection and programme effects, restrict effect heterogeneity
- typically hard to justify by economic reasoning
– Compare participants and nonparticipants that are identical for all
characteristics related to outcomes and participation (CIA; Rubin, 1974)
* confounders observed; instrument must exist, but is unobserved
+ nonparametric; justifiable by economic, institutional reasoning;
- very large and informative data necessary
– Find variable which affects outcomes only because it affects the
participation decision (IV, Imbens, Angrist, 1994, Heckman, 1978)
* instrument observed; some confounders unobserved
- average effect generally not identified
All nonparametric identification strategies
and related estimators belong to B or C!
(e.g. Vytlacil, 2002)

Credibility of identification strategies
– Depends on economic context (institutions, behaviour)
– Data
⇒ Get to know the context and the data first !

Switzerland provides an interesting example
– Active labour market policy
– Very good data
– Similar to other OECD countries
– Special: Considerable regional autonomy
– Special: Low level of unemployment

Switzerland and her 26 cantons

Expenditures in an international comparison
USA
Japan
Korea
Hungary
UK
Norw ay
Italy
Canada
Sw itzerland
Australia
Austria
Spain
France
Sw eden
Germany
Finland
Belgium
Netherlands
Expenditures for labour market policy
1999 in % of GDP
0 1 2 3 4
Passive measures
Active measures
source: OECD Employ ment Outlook 2003

Active labour market policy in Switzerland
Considerable active labour market policies
Expenditures for labour market policies in Switzerland (millions CHF)
6'000.0
5'000.0
4'000.0
3'000.0
2'000.0
1'000.0
-
1990 1995 1997 1998 1999 2000 2001
passive measures
Active measures

Cantonal variation in unemployment
Unemployment rates October 2003 by canton

- ORGANISATION of active labour market policy (ALMP)
Legislative authority: confederation.
Execution and implementation: cantonal
1997 / 1998 confederation wanted to make sure that cantons allocate many
UE into (new) programmes:
cantons which fill less than the required minimum number of year-places have to
compensate the federal unemployment insurance funds with 20% of the unemployment
benefits payments to those persons to whom no ALMP could be offered
– DESIGN of ALMP
- counselling, placement services
- training programmes and courses
- employment programmes
- subsidised interim jobs
- …
– RULES of ALMP
Willingness to participate is condition for receiving UE benefits
Caseworkers decide about participation

Data
Individual administrative records from unemployment
insurance system and social security (1988-1999)
identify outcomes, treatment status, region (municipality),
regional employment office in charge
Valuable individual information (proxies for motivation, etc.)
Regional information, commuting distances
identify local labour markets which have at least two parts
belonging to different administrative regions (cantons)

Structure of the discussion of empirical papers
A. What parameters do we want to know?
B. How do we identify them?
C. How do we estimate them?
D. What do we find?

Gerfin, Lechner (2002): Microeconometric Evaluation of Active
Labour Market Polices in Switzerland, Economic Journal
Gerfin, Lechner, Steiger (2003): Does subsidised temporary
employment get the unemployed back to work? An econometric
analysis of two different schemes, DP
A. What do we want to know ?
– Mean of effects of programmes for participants and nonparticipants
– Comparison of effects of different subprogrammes
– Mean effects for broad subgroups defined on observables
ATE, ATET, NATET (in subgroups)

Gerfin, Lechner (2002), Gerfin, Lechner, Steiger (2003)
B. How do we identify pairwise ATEs and ATETs?
Using good data to form comparison groups identical in all characteristics
influencing outcome and participation
(CIA: conditional independence assumption)
- labour market history over 10 years
- very detailed information about job before UE
- subject valuations of case workers
Still enough variation left - most likely unrelated to outcomes

Gerfin, Lechner (2002), Gerfin, Lechner, Steiger (2003)
C. How do we estimate pairwise ATEs and ATETs?
Matching on the propensity score on common support
(as in Lechner, 2002, REStat)
+ Simple and robust
- Not asymptotically efficient

Gerfin, Lechner (2002), Gerfin, Lechner, Steiger (2003)
D. What do we find?
ATET, ATE:
? Training not clear
+ Subsidised temporary jobs
- Employment programmes
Other findings:
- Mean effects vary not much with participation status
+ Bad risks may have a positive effect from a employment programme

Gerfin, Lechner (2002), Gerfin, Lechner, Steiger (2003)
Average effects for the population: A comparison to nonparticipation
A typical shape when programme duration is part of the effect

Gerfin, Lechner (2002), Gerfin, Lechner, Steiger (2003)
Average effects for the TEMP: A comparison to TEMP

Lechner, Smith (2003): What is the value added of case workers?,
DP
Frölich, Lechner, Steiger (2003): Statistically aided programme
selection, mimeo.
Ideas: I. Are case workers effective in programme selection?
II. If not, we help them by giving them our results ☺
A. What do we want to know ?
– Mean of effects of programmes for participants and nonparticipants for
different subgroups based on observables
– Comparison of effects of different subprogrammes
– Mean effects for broad subgroups defined on observables
ATE in small subgroups for predicting future outcomes

Lechner, Smith (2003), Frölich, Lechner, Steiger (2003) - SAPS
B. Identification? CIA as before
C. Estimation:
– Problem: Nonparametric estimates too imprecise in small cells
– (Semi-) parametric estimators based on the propensity score
– Results
- case workers as effective as random number generators
- considerable improvement possible if case workers use information about
effect heterogeneity coming from a system based on ‘individual’ predictions
of outcomes of different programmes
SAPS
statistically aided programme selection

Frölich, Lechner (2003): Regional treatment intensity
as instrument for the evaluation of active labour market policies
Idea
Use arbitrariness of regional boarders, regional autonomy and a ‘strange’
incentive introduced by the Swiss federal government as an instrument
A. What do we want to know ?
– Mean effects of programmes
– Can estimate what we want to know only for subpopulation reactive to
changes in instrument (compliers LATE, marginal effect)
– Note 1: Different instruments estimate may estimate different quantities
– Note 2: Some of those quantities are not interesting at all

Frölich, Lechner (2003) – regional IV
B. How do we identify LATEs?
– suppose people locate randomly in different parts of a region
– suppose this region constitutes one integrated labour market with two treatment regimes
– suppose that once being unemployed, the probability of participation in ALMP differs according to
specific part of region the UE is living
Living in a specific part of the region impacts participation probability, but has no impact on labour
market outcomes in the absence of participation
Location in region is a candidate for an INSTRUMENT!
Estimates mean effect for those who would change participation status if faced with different
participation probability (complier)!

Frölich, Lechner (2003) – regional IV
Integrated local labour markets
– About 170 regional employment offices (REO) in 1998
– definition of local labour market: Cluster of REO
– spread over 2 cantons
– commuting times between REO < 30 minutes (by car)
– same language (French, German or Italian)
– composition of ALMP is similar in REO on both sides
→ 30 integrated labour markets identified

Frölich, Lechner (2003) – regional IV
Conditions for valid exclusion restriction
A. Regional treatment intention (RTI) does not affect non-treated
and treated employment chances
B. RTI is independent of employability of local population
Conditions to estimate a mean causal effect for compliers
C. RTI is independent of type-composition of local population
D. Size of defier population is zero
E. Size of complier population is positive

Frölich, Lechner (2003) – regional IV
C. Estimation
Variation in participation due to diff‘s in quota is small weak instrument problem
1) Estimate LATE‘s for every pair of regions within the same local labour market
30 noisy estimates --> aggregate!
[ = ] − [ = ]
[ | ''] [ | ']
1 0
EY| Z z'' EY| Z z'
E⎡
⎣Y −Y | Τ= c⎤
⎦ =
E D Z = z − E D Z = z
instrument weak denominator small
Wald (IV, 2SLS, LIML, …) estimator has bad small sample properties
1
Use fuller estimators instead: ˆ α
−
θ = ⎡ '( (1 − ) + ) ⎤ '( (1 − ) + )
⎣D I k kP D ⎦ D I k kP Y
' −1 '
PN = ZN( ZNZN)
Z
α
N k = k LIML
−
N − L
N N N N N N N N

Frölich, Lechner (2003)-IV
D. Empirical results: Regions with similar ALMP
Employment (in%-points) High earnings (in %-p.)
May ‘98 Sept May Sept.’98
Effect 9.5 8.3 13.2 10.4
Std. error 7.1 7.0 7.0 7.3
t-value 1.33 1.19 1.88 1.43
Bootstrapped t-distribution
Q 0.95
1.29 1.47 1.13 1.23
Q 0.975
1.60 1.72 1.35 1.59
Q 0.995
1.98 2.26 1.67 2.22

Lechner, Miquel (2001): Identification of dynamic effects, DP
Lechner (2004): Dynamic matching, DP University of St. Gallen
Miquel (2003a): Dynamic diff-in-diff, DP University of St. Gallen
Miquel (2003b): Dynamic IV, DP University of St. Gallen
Idea of I and II: Interest is in effects of sequences of treatment when data is good
enough to control for sequential selection biases
Idea of III and IV: Instruments
A. What do we want to know and what is the problem?
– Mean effects of sequences of programme participation (DATE, DATET)
– Sequential selection
– Selection may be based on endogenous variables (intermediate outcomes)

Lechner, Miquel (2003), Lechner (2003) - dynamics
B. Identification?
– DCIA, weak and strong versions depending on how selection process works
– W-DCIA does identify DATE, but not all DATET
– Estimation
- Sequential matching
D. Results
- not yet to be presented

Part II:
Standard estimators used in policy evaluation
OLS, GLS, Probit, Duration models etc.
- CIA combined with functional form assumptions
- effect homogeneity
- need very informative data for CIA to hold
Matching (many ways to do it)
- nonparametric version of above; CIA
- need very informative and large data

Part II: Standard estimators used in policy evaluation
Linear IV (SLS. etc.)
- exploits exclusion restriction combined with functional form
- effect homogeneity
Nonparametric IV with discrete instruments
- no functional form necessary
- exclusion restriction often hard to justify
- instruments may often be weak
- only LATE is identified

Part II: Standard estimators used in policy evaluation
Before-after, difference in difference, fixed effects, etc.
- exploits information for the participants before participation
- exploits information for the nonparticipants before and after participation
- effect homogeneity (within: strict exogeneity)
- even ‘nonparametric’ version requires functional form assumptions (e.g. required
assumption may hold in levels but not in log’s etc.)
- having a panel to analyse does not mean that typical panel estimators must be used
(matching or standard IV may be superior)
Control function estimators (Heckit, other model with ‘unobserved
heterogeneity’)
- nonparametric version identical to IV (+functional forms)
- even parametric version needs exclusion restriction to produce stable estimates

Part II: Standard estimators used in policy evaluation
Regression discontinuity design
- useful in cases when all are subject to some policy once a threshold is passed
no ready comparison groups, because no support
- local IV (passing threshold has no effect!)
- local LATE
Every other econometric estimator technique has an interpretation
as an estimator for causal effects
But: Implicit conditions for identification of causal effects must be derived. Causal effect
may not be readily seen by looking at coefficient.
Structural modelling
Partial equilibrium economic choice models
CGE’s
Again: Are the underlying assumptions plausible? What are the economic / behavioural
restrictions implied by ‘statistical assumptions’?

Are some methods magic bullets ?
Important issues in performing an evaluation study:
– Know the selection mechanism and how the programmes are run
– Derive restrictions from the selection mechanism
– Data collection: Case 1 - you have no influence on data collection:
- given the data available, is there a plausible identification strategy?
- if so, are there enough observations to use a nonparametric estimator?
- would a structural modelling approach be plausible?
– Data collection: Case 2 – you have influence on the data collection:
- make sure the data is collected such that reasonable identification
strategy becomes feasible
- make sure sample is large enough so that nonparametric estimation
methods can be used

Are some methods magic bullets ?
The merits of any of the discussed evaluation
strategies depends entirely on the institutional
background, on the behaviour of the agents in the
selection and outcome processes, and on the data
available.
There is no evaluation method that
is superior in all circumstances!