Download PDF - SRI International

As with any type of effectiveness study, estimating outcomes for productivity need to follow
commonly accepted guidelines for program evaluation (e.g., see Frechtling and Sharp 1997).
The outcomes selected to measure and report should be the most important to the
intervention and attributable specifically to the program. For example, if the program is a
single mathematics class that students can take either face to face or online, gains in
mathematics test scores for participating students at the end of the year could be evaluated;
however, it may be difficult to draw clear connections between a specific program and larger
academic outcomes (e.g., graduation) that could be influenced by a host of factors outside
the scope of the mathematics class. A study also may take into account multiple outcomes as
is common in cost-benefit or cost-utility studies (Levin and McEwan 2001).
Cost-Effectiveness Research Requirements
The above productivity framework suggests a number of requirements for sound studies of
online learning productivity that are offered here to help guide literature analysis.
1. Specify important design components of the intervention. Because the costs and
outcomes of different programs can vary widely, simple comparison of ratios will do
little to elucidate the factors that contributed to productivity gains. For study results
to suggest design features worthy of replication, the study must describe important
variations between the treatment and control conditions. In some studies, the only
salient differences are related to technology and delivery systems. However, welldesigned
online learning interventions generally include modifications in pedagogy
and curricular materials and other enhancements that take advantage of the
technology platform. These factors may not be included in a productivity ratio, but
including them in research reports is important for supporting the interpretation of
reported productivity ratios.
2. Compare at least two conditions. On its own, a ratio of per-pupil program cost per
unit of outcome is not meaningful. An online and control condition (e.g., comparing
the ratio of costs and test scores in an online academy with that in a brick-and-mortar
school teaching the same content) can be used to measure changes in productivity
ratio across conditions. Analysts should also consider blended alternatives, not just
contrasts between fully online learning and fully face-to-face instruction.
3. Measure both costs and outcomes. Both costs and outcomes can vary widely across
implementations, suggesting that both factors must be measured for each
intervention under study. The study should use the same cost framework for the two
conditions so that all relevant costs—and the same costs—are compared. Similarly,
12