MONITORING AND EVALUATION FRAMEWORK ... - The Presidency

MONITORING AND EVALUATION FRAMEWORK
(Implications for Government)
Presented at the Senior Management Service Conference
by
Kefiloe Masiteng
21 September 2004

INTRODUCTION
MANDATE
• Monitoring and evaluation of
government performance has
been identified as a
responsibility of the PCAS
• The Presidency may evaluate the
performance of government
against set goals, targets,
equitableness of resource
allocation and effectiveness and
efficiency in service delivery
across all levels.
OBJECTIVE
To inform steps to be
undertaken in creating a
conducive environment enabling
all levels of government, private
sector, communities and
individuals to achieve respective
goals in service delivery and
Improve performance.
Hence the need to assess essential capacity in govt.

Monitoring versus Evaluation
• MONITORING
Tracking changes in program performance
over time
• EVALUATION
Attributing program outcomes to their
causes

MONITORING
WHY MONITOR
• Monitoring in the government-wide
framework refers to a set of activity
and milestone tracing techniques, all
of which measure some aspect of
government performance including
the measurement of the current
status and change over time (trend
analysis) in any of the initiatives.
• Monitoring tracks changes in
services provided (outputs) and the
desired results (outcomes), providing
the basis for accountability in the
utilization of resources.
BENEFITS
• Monitoring can be put into
place as a management tool
that may be sustained over
time. It can be used to improve
initiatives by identifying
aspects that are working
according to plan and yielding
positive results, while on the
other hand it can identify those
initiatives that need midcourse
corrections.
POA: Monitoring of the progress made in
attaining the goals set in the SONA and Makgotla.
Cluster POAs: Bi-monthly reporting to Cabinet

MONITORING C0MPONENTS
Monitoring processes:
• Development and definition of indicators
to measure the progress made towards
meeting relevant objectives
• Data collection mechanisms for and
monitoring systems to collate data on
indicators
• Data verification, validation and systems
clean up
• Data analysis to determine outputs,
outcomes and trends
• Report writing on the progress made on
implementation
• Distribution and feedback mechanisms
across the entire spectrum of relevant
stakeholders.
Capacity needed for Govt:
• Understanding of the
POA on GOVT. website
• Ability to develop
relevant indicators for
the initiatives and
interventions arising
from the cluster POA
• Information collection
strategy on the
developed indicators
• Analysis and verification
of collected information
• Report writing
• Communication link with
GCIS

DEVELOPMENT AND DEFINITION OF
RELEVANT INDICATORS
• Indicators development is based
on goals and objectives for
government
• These indicators may be
calculated on the basis of
description and formulae allocated
to measure progress made
(monitoring) or determining
causality (impact assessment).
– method for acquiring information
on indicators,
– responsibility for collection
– Info./data source
– frequency for updating
• Agreement on evaluation methods
• Role of Presidency
• Spearheading the
indicator development
based on POA
• Setting in place
collection, collation and
report-back/feedback
mechanisms

IMPACT ASSESSMENT
• The purpose of impact assessment
is primarily to measure the degree of
change attributable to a particular
initiative or intervention.
• Impact assessment addresses the
question of causality.
• What differentiates the two
processes are the evaluation
techniques which might just include
trend analysis in the case of
monitoring and the analytic
techniques used in impact
assessment.
• It determines how much of the
observed change in the outcome
(quality of life, access to services
e.t.c) at the population can be
attributable directly to the
implementation of government
policies and programmes and not to
other factors.
• The level of analysis for assessing
the impact of government policies
and programmes is the population
(beneficiaries).
•Lessons from the Ten Year Review are crucial
•Planning for future government reviews based on TYR indicators
•Development of Mid-term review indicators based on MTSF
•Review and refinement of current TYR indicators

IMPACT ASSESSMENT
• Development of assessment
frameworks (modeling)
• Collection and collation of data
from different sources in relation
to developed models
• Regression (logistic, multivariate
e.t.c) analysis on dependent and
independent variables
• Interpretation of results/findings to
determine relationships
• Report writing on the impact of
government interventions to the
population
• Distribution of reports to relevant
stakeholders
PROCESSES
Implications for Presidency
• Advanced policy analysis
skills
• Advanced data analysis
skills
• Basic data mining
Basic statistical modeling skills
• Econometrics
• Demographic modeling
(work with treasury on economic models)
(Working with departments to compile a compendium of indicators)
(work with Statssa on demographic/population dynamics & NSS)

DEVELOPMENT OF DATA COLLECTION
MECHANISMS FOR INDICATORS
• To enable comparisons (demographic,
social, economic, financial and
corporate governance) across the
provinces, population groups, gender
and age groups around government
sectors over time and space.
• The data collected on indicators will
thus have to accommodate such
comparisons and be disintegrated
within the developed systems and
databases according to the above
mentioned categories, especially the
GDC.
• Role of Government
– Data collection
– Verification
– Validation
– Report writing

INFORMATION MANAGEMENT
• The use of information systems in
monitoring provides a reliable flow
of information to allow
management to keep abreast with
the progress in the
implementation of policy thrusts,
programmes and activities based
on decisions made in different
gatherings.
• Information systems facilitate
assessment of the quality,
quantity and timeliness of policy
and programme inputs while
operational constraints towards
programme and policy
effectiveness are identified, thus
gaps may be addressed.
• They may further provide
contextual information for
evaluation processes.
Role of Presidency
• Reporting Formats
from FOSAD
– (the project card)
• EIMS
– Roll out
– TRAINING
– Commitment
• Integration with NSS
(urgent to review)

Three models may be applied in monitoring
and evaluation activities in Government
• High level tertiary model: This model can be informed by State
of the Nation address, Cabinet Decision and cluster priorities;
• Government level monitoring and evaluation (PCAS): This
model measures the progress made by government as a whole
in addressing the objectives and implementing priority
programmes;
• Departmental Monitoring and Evaluation Initiatives:
This level addresses the progress made by individual
departments in implementing their programmes in line with
government priorities. These include indicators to measure
programme level objectives (outputs), developed within each
department in their informed by their strategic frameworks

Government model for systems
integration at National level
Govt.
POA
Executive
Info.
(CabEnet)
Planning
Government
Statistics
(Departmental routine
sy stems)
National Statistical System)
M&E
Programme level Statistics/Info
(Departmental Information Systems)
Provincial and Local Government M&E

OPERATIONALISING GOVERNMENT- WIDE
MONITORING AND EVALUATION (PHASE 1)
Five results, each of which will be delivered as a Report
• A review of existing public service monitoring and evaluation systems.
An early step in the process of creating a national monitoring and evaluation
system for government will involve reviewing existing Departmental M&E
systems so that existing capacity and capability is properly drawn upon.
• A review of government reporting requirements, procedures and needs.
• A review on progress in the development and implementation of government wide
M&E systems by central or coordinating departments.
• Results of consultations with all provincial administrations and FOSAD clusters on
their performance indicators.
• A logic model and framework architecture for a national M&E framework, (including
a dashboard-style presentation of a national scorecard).

Monitoring and Evaluation
Refining Monitoring and Evaluation system
Improve quality of our outputs
Provide an early warning system
Improve statistical and information base.
Cabinet
Clusters and Depts
FOSAD & MANCO?
Project Teams
Presidency

Monitoring and Evaluation
Government's Programme of Action
Econom ic Cluster
FIRST ECONOMY
In dex | Next
Actions Departments Tim eframe Progress
1.00 BROAD ECONOMIC STANCE
1.01 N o significant cha nges to m acroeconom ic
m anag em ent foreseen .
NT Ong oing Continuing
1.02 In fla tion targeting to re m ain in pla ce NT Ong oing Continuing
1.03 The Execu tive sho uld work w ith the m onetary
authorities to ensure th at the objective of a sta ble
and com petitive exch ange rate and approp riate
inflation range is attained
NT Ong oing Con tinu ing thro ugh
ordinary cha nnels
1.04 Shift to m icro eco nom ic re form s dti, N T,
Presid ency
Ong oing
Ongoin g
2.00 INCREASED LEVELS OF INVESTM ENT IN
PUBLIC INFRASTRUCTURE
2.01 D eve lop sector-specific infrastructure in vestm ent
plans
2.02 The developm ent of financing stra teg ies to
im plem en t plans including Public Private
Partnership s and sim ilar m echanism
2.03 Im p lem entatio n o f an appro priate regulatory
fram ew ork to stim ulate new S tate Ow ned
Enterprise investm en t in infrastructure
DPE, DM E, d ti,
NT, DOT
DPE, DM E, d ti,
NT, DOT
DPE, NT,
Presid ency
2.04 Strengthenin g o f sector regu lato rs DPE, DM E,
DOC
N ov-04
Sep -04
Sep -04
C ontinuous
Progre ssing
Progre ssing
Progre ssing
2.05 Strengthenin g o f th e G ove rnm ent cap acity to
oversee the State O wn ed Ente rprises
DPE, NT,
Presid ency
C ontinuous
2.06 Finalise C AP EX fin ancing strategy DPE et al Sep -04 Progre ssing
2.07 Finalise G overnm ent-w ide review of the State
Ow ned Enterprises perform an ce pra ctices
2.08 B ette r utilis ation of P ub lic Inves tm ent
C om m ission 's Isibaya fun d
DPE, NT,
DPSA
Sep -04
Draft policy to be
com p lete d by N ovem ber
2004
Progre ssing
NT, GDC N ov-04 Progre ssing
2.09 Expedite Port re structuring DOT, DPE a.s.a.p B ill to be tab led in N C O P
by Septem ber 200 4

Monitoring and Evaluation
PROGRAMME OF ACTION
Governance and Administration (G&A Cluster)
Policy Objective 4.00 - MACRO-ORGANISATION OF THE ST ATE
Program me Activity 4.04.2 - Build 60 Multi Purpose Community Centres (MPCCs) and finalise plans to
have at least one of these in each of our 284 municipal areas
Project Team: GCIS
People’s Contract/Partnerships:
Timeframes: Dec-04
WORK IN PROGRESS
Strategic Policy Issues:
Performance Indic ators:
Level of Progress:
Fast Tracking Implementation:
Challenges:
Comments:

ROLE OF COORDINATING
DEPARTMENTS
Institutions at the centre of government need to take the initiative in designing
performance assessment systems for the whole of government i.e PCAS, OPSC,
National Treasury, DPSA, DPLG.
These should link clearly into the Medium Term Expenditure and Strategic Frameworks
and should show how assessments and evaluations should deliver useful information
with practical recommendations.
Such transversal systems could include:
• Good governance (OPSC/Presidency)
• Value for Money (National treasury)
• Service Delivery (DPLG/OPSC)
• Human Resource utilization (DPSA)
• An Early Warning Systems (DPSA/Presidency)

ROLE OF SECTOR DEPARTMENTS
Government wide M&E system will be operationalised on the
understanding that each individual department will take responsibility for
their own monitoring and evaluation processes according to the
guidelines and standards mentioned above.
• Monitoring is meant to take place at three different levels:
‣ Implementation monitoring, evaluation, early warning and data collection – at all
three spheres of government using inputs, outputs and outcome indicators
‣ Monitoring of national departmental inputs, outputs and outcomes – by the
coordinating departments (PCAS, OPSC, National Treasury, DPSA, DPLG)
‣ Monitoring of process inputs, outputs and outcomes – by the departments
themselves
• Evaluation will also take place at these three levels but will be restricted to
process and impact analysis.

ROLE OF PROVINCIAL AND LOCAL
GOVERNMNET
• Operationalisation of the framework will comprise provincial, departmental systems
and the government wide supplementary systems listed above, some of which still
need to be developed. Work on such development should be considered a priority.
• Government wide M&E system will be operationalised on the understanding
that individual Provinces will take responsibility for their own monitoring and
evaluation processes according to the guidelines and standards mentioned
above.
• The role of Premier’s Offices in driving provincial M&E will also need a
special focus. This highlights the need for the offices of the Premiers in all
Provinces to establish monitoring and evaluation processes and apply them
to local government.

Components of Programme Monitoring and
Impact Assessment
Program level
Population level
INPUTS PROCESSES OUTPUTS OUTCOMES
Resources:
•Personnel
•Equipment
•Finance
Project Cycle: Phases
1. Housing Dev. Process
•Access to land
•Land avail. Agreement
2. Planning Process
•Layout
•Civil eng. Design
3. Township est. process
•Install Civ. Eng. Services
•Units construction
4. Hand over process
•Keys to beneficiaries
Deliverables
•Serviced sites
•Subsidies approved
•Units completed
•Units under const.
•Projects approved
•Fem. Headed H/holds
•Budget & Exp.
Impact
•Housing access
•Better lives
• % Beneficiaries
•Objectives met
Monitoring
Impact
Assessment

RELEVANT SKILLS AND RESOURCES
REQUIRED
• Research;
• statistical/ data analysis;
• specialized software to perform modeling and
other evaluation techniques;
• Research design for evaluation may include
population surveys, community surveys or
forums, focus groups as well as randomized
experiments;
• Policy analysis and report writing.

Key Questions for Program M&E
• What is a program?
• Nationally organized, often
publicly sponsored, effort to
deliver social-economic
services to target
populations with need
• Organizational systems
activated for service
delivery
• Indefinite lifetime
• Has an institutional host
that is organic, of known
size, adaptive, and
operates in a changing
environment
• Did the program achieve
its objectives?
• Were the results
attributable to program
efforts?
• Which program activities
were more or less
important/effective?
• Did the intended benefit
from the program?
• At what cost?

Scope of Program M&E
• What level of program evaluation?
• National, subnational, specific site?
• Implications for M&E design
• Inference of results
• Relevant time frame?
• Relevant units of action?

RELEVANT SKILLS AND RESOURCES
REQUIRED
• Research;
• statistical/ data analysis;
• specialized software to perform modeling and
other evaluation techniques;
• Research design for evaluation may include
population surveys, community surveys or
forums, focus groups as well as randomized
experiments;
• Policy analysis and report writing.

Illustration of Program Monitoring
Program
outcome
indicator
Program
start
TIME->
Program
end

Illustration of Program Monitoring
Program
outcome
indicator
Actual?
Program
start
TIME->
Program
end

Illustration of Program Impact
With program
Change
in
program
outcome
Without
program
Program
start
TIME->
Program
end

Illustration of Program Impact
Change
in
program
outcome
Program
impact
With program
Without
program
Program
start
TIME->
Program
end

The Role of the
Logical/Strategic/Conceptual Framework
• Logical vs Strategic vs Conceptual
• Clarify program objective/strategic
outcome/dependent variable
• Interrelate units, levels and directions of
action
• Allow for consensus-building around a
common paradigm

Example of a Strategic Framework
Strategic Objective/Priority
Objective 1 Objective 2
Indicator 1
Indicator 2
Indicator 1
Indicator 2
Indicator 3

Example of a Conceptual Framework
for a Structural Model
Individual
demand
Program
supply
Service
utilization
Output of
delivery
Adequacy of
Delivery

Example of a Conceptual Framework
a Structural Model
Individual
demand
Program
supply
Technical
inputs
Service
utilization
Housing
Delivery
Institutional
capacity
Adequate
Housing
Selfsufficiency

Monitoring versus Evaluation
Can good monitoring lead to good evaluation?
• Can good monitoring lead to
good evaluation?
• Indicators = Significant and
influential factors
• Framework = Theoretically
sound model
• Directionality = Temporally
correct causal flow
• Coupling
quantitative
and
qualitative
assessment
methods
• Levels = Appropriate hierarchy
of units