Primary Education Survey Evaluation Report Somalia - Somali - JNA

PES Evaluation Report, 2008
1 Validity
1.1 Findings and Analysis
Attempts were made to assess data quality. Unfortunately this was undertaken without a full field
assessment being conducted by selecting a sample of schools and recollecting data for a comparison. This
full validation exercise was impossible due to schools being closed for summer holidays. Such an
evaluation was also outside of the terms of reference for this study. Instead a range of participants in the
process were interviewed and an assessment made of documentary evidence, processes in the project
design, and comparison with stakeholders perceptions based on field experience.
There has been some questioning of the validity of some of the data that appears on the data forms and in
the draft report of the 2006/7 survey. Comment has been made on the draft report in the reporting section
of this evaluation. There it is argued that the inaccuracies and inconsistencies identified seem to be a
function of the subsequent analysis of the data and careless editing rather than a consequence of the lack
of validity in the original data set. That is not to imply that there are no limitations on the validity of the
data in the data base. The issue is to assess the size and nature of the limitations and to recommend
strategies to address the threats to validity. As one Director General interviewed observed:
All measurement has an error. But it is important to improve results.
In attempts to reduce the size of the error the issue of diminishing returns for effort is faced. By this it is
meant that it is relatively cheap to move from 90% accurate to 91% accurate, but very expensive to move
from 98% accurate to 99% accurate, with 100% accuracy being unattainable. This is especially so in a
context of political instability, capacity constraints, rainy seasons, droughts and a mobile population.
There are a number of reported threats to the validity of the data, some of which are of the nature of
hunches, and some of which are a result of direct observation. An interview with a director general
identified these issues:
A number of schools were not visited and some documents lost, but these matters have
been discussed and follow up data collection conducted to remedy the situation.
The recording of GPS coordinates is a problem and I suggest that additional training on
the use of a GPS is necessary.
The process of transferring raw data to UNICEF for data entry caused some problems,
and to ensure supervisors and enumerators completed forms correctly we need close
monitoring. (He was unable to tell us about spot checking).
In one zone UNICEF officers make up to 150 visits to schools (out of approximately 600 schools) for
check visits. One reported that only 1 out of his 30 schools had not been visited. In another zone the
UNICEF officer said that in 4 years he had only found one school that had not been visited.
The Zone that had the trouble with the use of the GPS and recording the coordinates was also the zone
that did not conduct field testing during the training of enumerators. In another zone that had conducted
field testing
Further:
Only one enumerator had trouble with GPS. The problem was identified and resurveyed.
We had a problem with School mapping data last year. The coordinates collected did not
match the ten year old village location coordinates. We recollected 10 Bossaso school
coordinates and they still did not match. Was the problem with us or the village maps?
14