Gibson Ferguson Language Planning and Education Edinburgh ...

Educational and political dimensions of bilingual education 53
eventually identifying only 11 as methodologically acceptable. Excluded were studies
that could not be located (5), that did not evaluate bilingual programs (3), that did
not properly control for background differences between treatment and control
groups (25), that did not compare programs taught entirely in English with those
including some native language (L1) instruction (14), and that were re-analyses of
the same program by the same authors (15). The remaining 11 were submitted to
meta-analysis to take into account treatment effect sizes in each study, and on this
basis Greene (1997: 6) concluded, contrary to Rossell and Baker, that native
language instruction had positive effects on English test results and some less certain
benefits for mathematics scores.
In acknowledgment of the limited number of high quality studies, Greene (1997:
11) makes a final plea for the commissioning of more ‘random-assignment
experiments … to compare different approaches to teaching LEP students’.
However, for reasons we now come to, many commentators would be most reluctant
to respond to this call.
3.2.2 Theory and the evaluation of bilingual education programs
Policy-makers who turn to formal evaluations of bilingual education (BE) programs
for guidance are likely to experience disappointment. Filtered through the disputes
of ideological adversaries, the findings appear to be, and are, inconclusive and
inconsistent. In part, this is attributable to the formidable methodological obstacles
to any quasi-experimental comparative evaluation of types of educational program,
the most serious of which (see Willig and Ramirez 1993) is that the traditional best
method of establishing true comparability between students in treatment and
control groups, random allocation, is rarely feasible in the educational field. Other
difficulties include changes in the composition of groups as pupils exit and join
programs, and the ascertaining of a match between a program’s label and its actual
implementation. Program evaluations have also been criticised (Baker 2001) for
focusing unduly on easily measurable attainments and overlooking less easy to
ascertain but important outcomes such as individual self-esteem, emotional stability,
parental involvement in schooling and future earnings.
Apart from the methodological concerns, the perception has also grown that
program evaluations have been driven by the wrong kind of question. It is now
widely acknowledged that programs are not single entities (see August and Hakuta
1997: 19) but complex constellations of interacting components, and that therefore
the more productive question is not ‘which program is best for LEP pupils?’ but
rather which program components and ‘which sets of instructional practices allow
identified groups of English learners (LEP pupils) to reach educational parity, across
the curriculum, with the local or national group of native speakers of English?’
(Thomas and Collier 1997: 19).
The consequence has been not so much an abandonment of comparative program
evaluation (see Thomas and Collier 1997) but a reduced preoccupation with the
single variable of language of instruction and a renewed emphasis on identifying