The Effects of Distance Education on K-12 Student Outcomes: A ...
The Effects of Distance Education on K-12 Student Outcomes: A ...
The Effects of Distance Education on K-12 Student Outcomes: A ...
Create successful ePaper yourself
Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.
<str<strong>on</strong>g>The</str<strong>on</strong>g> <str<strong>on</strong>g>Effects</str<strong>on</strong>g> <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Distance</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g> <strong>on</strong> K-<strong>12</strong> <strong>Student</strong> <strong>Outcomes</strong>: A Meta-Analysis<br />
Cathy Cavanaugh, University <str<strong>on</strong>g>of</str<strong>on</strong>g> North Florida<br />
Kathy Jo Gillan, Duval County Public Schools<br />
Jeff Kromrey, University <str<strong>on</strong>g>of</str<strong>on</strong>g> South Florida<br />
Melinda Hess, University <str<strong>on</strong>g>of</str<strong>on</strong>g> South Florida<br />
Robert Blomeyer, North Central Regi<strong>on</strong>al <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>al Laboratory<br />
This study was supported by the North Central Regi<strong>on</strong>al <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>al Laboratory (NCREL). <str<strong>on</strong>g>The</str<strong>on</strong>g><br />
c<strong>on</strong>tent does not necessarily reflect the positi<strong>on</strong> or policy <str<strong>on</strong>g>of</str<strong>on</strong>g> NCREL.<br />
Corresp<strong>on</strong>dence c<strong>on</strong>cerning this paper should be addressed to Cathy Cavanaugh, Curriculum and<br />
Instructi<strong>on</strong>, University <str<strong>on</strong>g>of</str<strong>on</strong>g> North Florida, Jacks<strong>on</strong>ville, FL 32224. ccavanau@unf.edu<br />
1 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
Abstract<br />
Since the early 1930s, elementary and sec<strong>on</strong>dary students have learned through the use <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
electr<strong>on</strong>ic distance learning systems. Several benefits have been reported for K-<strong>12</strong> distance<br />
educati<strong>on</strong>: increased access to educati<strong>on</strong> for students with a wide range <str<strong>on</strong>g>of</str<strong>on</strong>g> needs, flexibility in<br />
the speed and schedule <str<strong>on</strong>g>of</str<strong>on</strong>g> learning, greater parental influence <strong>on</strong> educati<strong>on</strong>. However, some<br />
researchers have found that the effectiveness <str<strong>on</strong>g>of</str<strong>on</strong>g> distance educati<strong>on</strong> depends <strong>on</strong> the c<strong>on</strong>texts in<br />
which it occurs. <str<strong>on</strong>g>The</str<strong>on</strong>g> research <strong>on</strong> the effects <str<strong>on</strong>g>of</str<strong>on</strong>g> distance educati<strong>on</strong> <strong>on</strong> K-<strong>12</strong> student outcomes has<br />
been somewhat ambiguous. This meta-analysis <str<strong>on</strong>g>of</str<strong>on</strong>g> 116 effect sizes from 14 web-delivered<br />
distance educati<strong>on</strong> programs studied between 1999 and 2004 shows that distance educati<strong>on</strong> can<br />
have the same effect <strong>on</strong> measures <str<strong>on</strong>g>of</str<strong>on</strong>g> student academic achievement when compared to traditi<strong>on</strong>al<br />
instructi<strong>on</strong>. <str<strong>on</strong>g>The</str<strong>on</strong>g> study-weighted mean effect size across all outcomes was -0.028 with a 95%<br />
c<strong>on</strong>fidence interval from 0.060 to -0.116. No factors were found to be related to significant<br />
positive or negative effects. <str<strong>on</strong>g>The</str<strong>on</strong>g> factors that were tested included academic c<strong>on</strong>tent area, grade<br />
level <str<strong>on</strong>g>of</str<strong>on</strong>g> the students, role <str<strong>on</strong>g>of</str<strong>on</strong>g> the distance learning program, role <str<strong>on</strong>g>of</str<strong>on</strong>g> the instructor, length <str<strong>on</strong>g>of</str<strong>on</strong>g> the<br />
program, type <str<strong>on</strong>g>of</str<strong>on</strong>g> school, frequency <str<strong>on</strong>g>of</str<strong>on</strong>g> the distance learning experience, pacing <str<strong>on</strong>g>of</str<strong>on</strong>g> instructi<strong>on</strong>,<br />
timing <str<strong>on</strong>g>of</str<strong>on</strong>g> instructi<strong>on</strong>, instructor preparati<strong>on</strong> and experience in distance educati<strong>on</strong>, and the setting<br />
<str<strong>on</strong>g>of</str<strong>on</strong>g> the students.<br />
2 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
<str<strong>on</strong>g>The</str<strong>on</strong>g> <str<strong>on</strong>g>Effects</str<strong>on</strong>g> <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Distance</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g> <strong>on</strong> K-<strong>12</strong> <strong>Student</strong> <strong>Outcomes</strong>: A Meta-Analysis<br />
Beginning in the 1930s, radio was used simultaneously to bring courses to school students and to<br />
help teachers learn progressive Deweyan methods <str<strong>on</strong>g>of</str<strong>on</strong>g> teaching (Bianchi 2002), in what might<br />
have been am<strong>on</strong>g the earliest pr<str<strong>on</strong>g>of</str<strong>on</strong>g>essi<strong>on</strong>al development school models. From that point <strong>on</strong>,<br />
televisi<strong>on</strong>, audio and videoc<strong>on</strong>ferencing, the Internet, and other technologies have been adapted<br />
for the needs <str<strong>on</strong>g>of</str<strong>on</strong>g> young learners.<br />
<str<strong>on</strong>g>Distance</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g> in the K-<strong>12</strong> C<strong>on</strong>text<br />
Am<strong>on</strong>g the benefits <str<strong>on</strong>g>of</str<strong>on</strong>g> distance educati<strong>on</strong> for school-age children are increases in enrollment or<br />
time in school as educati<strong>on</strong> programs reach underserved regi<strong>on</strong>s, broader educati<strong>on</strong>al opportunity<br />
for students who are unable to attend traditi<strong>on</strong>al schools, access to resources and instructors not<br />
locally available, and increases in student-teacher communicati<strong>on</strong>. <strong>Student</strong>s in virtual schools<br />
showed greater improvement that their c<strong>on</strong>venti<strong>on</strong>al school counterparts in critical thinking,<br />
researching, using computers, learning independently, problem-solving, creative thinking,<br />
decisi<strong>on</strong>-making, and time management (Barker & Wendel 2001). Academic advantages over<br />
traditi<strong>on</strong>al classroom instructi<strong>on</strong> were dem<strong>on</strong>strated by students in Mexico’s Telesecundaria<br />
program, who were “substantially more likely than other groups to pass a final 9 th grade<br />
examinati<strong>on</strong>” administered by the state (Calder<strong>on</strong>i, 1998, p. 6); by students taking a chemistry by<br />
satellite course (Dees 1994); and by students learning reading and math via interactive radio<br />
instructi<strong>on</strong> (Yasin & Luberisse 1998).<br />
<str<strong>on</strong>g>Distance</str<strong>on</strong>g> educati<strong>on</strong> is not the most effective choice in all situati<strong>on</strong>s. <strong>Student</strong>s may feel isolated,<br />
parents may have c<strong>on</strong>cerns about children’s social development, students with language<br />
difficulties may experience a disadvantage in a text-heavy <strong>on</strong>line envir<strong>on</strong>ment, and subjects<br />
requiring physical dem<strong>on</strong>strati<strong>on</strong>s <str<strong>on</strong>g>of</str<strong>on</strong>g> skill such as music, physical educati<strong>on</strong>, or foreign language<br />
may not be practical in a technology-mediated setting. For example, B<strong>on</strong>d (2002) found that<br />
distance between tutor and learner in an <strong>on</strong>line instrumental music program has negative effects<br />
<strong>on</strong> performance quality, student engagement, and development and refinement <str<strong>on</strong>g>of</str<strong>on</strong>g> skills and<br />
knowledge. While distance learning was viewed as beneficial for providing the opportunity for<br />
elementary school students to learn a foreign language, C<strong>on</strong>zemius and Sandrock (2003) report<br />
that “the optimal learning situati<strong>on</strong> still involves the physical presence <str<strong>on</strong>g>of</str<strong>on</strong>g> a teacher” (p. 47).<br />
Virtual school students show less improvement than those in c<strong>on</strong>venti<strong>on</strong>al schools in listening<br />
and speaking skills (Barker & Wendel 2001). Highly technical subjects have also proven to be<br />
difficult to teach well <strong>on</strong>line. <str<strong>on</strong>g>The</str<strong>on</strong>g> Alberta Online C<strong>on</strong>sortium evaluated student performance <strong>on</strong><br />
end-<str<strong>on</strong>g>of</str<strong>on</strong>g>-year exams am<strong>on</strong>g virtual school students across the province, and found that virtual<br />
school student scores in mathematics at grades 3, 6, 9, and <strong>12</strong>, and the sciences at grades 6 and 9<br />
lagged significantly behind scores <str<strong>on</strong>g>of</str<strong>on</strong>g> n<strong>on</strong>-virtual school students (Schollie 2001).<br />
Given instructi<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> equal quality, groups <str<strong>on</strong>g>of</str<strong>on</strong>g> students learning <strong>on</strong>line generally achieve at levels<br />
equal to their peers in classrooms (Kearsley 2000). Equality between the delivery systems has<br />
been well documented over decades for adult learners, and while much less research exists<br />
focusing <strong>on</strong> K-<strong>12</strong> learners, the results tend to agree. “Evidence to date c<strong>on</strong>vincingly<br />
dem<strong>on</strong>strates that , when used appropriately, electr<strong>on</strong>ically delivered educati<strong>on</strong>—‘e-learning’—<br />
3 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
can improve how students learn, can improve what students learn, and can deliver high-quality<br />
learning opportunities to all children” (NASBE 2001, p. 4). Many studies report no significant<br />
differences between K-<strong>12</strong> distance educati<strong>on</strong> and traditi<strong>on</strong>al educati<strong>on</strong> in academic achievement<br />
(Falck et al 1997, Goc Karp & Woods 2003, Hinnant 1994, Jordan 2002, Kozma et al 2000,<br />
Mills 2002, Ryan 1996), frequency <str<strong>on</strong>g>of</str<strong>on</strong>g> communicati<strong>on</strong> between students and teachers (Kozma et<br />
al), and attitude toward courses (McGreal 1994).<br />
Although technology-enabled distance educati<strong>on</strong> for pre-college students is nearing its century<br />
mark, comparatively little research has been published that can serve to guide instructors,<br />
planners, or developers. <str<strong>on</strong>g>The</str<strong>on</strong>g> temptati<strong>on</strong> may be to attempt to apply or adapt findings from<br />
studies <str<strong>on</strong>g>of</str<strong>on</strong>g> K-<strong>12</strong> classroom learning or <strong>on</strong> adult distance learning, but K-<strong>12</strong> distance educati<strong>on</strong> is<br />
fundamentally unique.<br />
A primary characteristic that sets successful distance learners apart from their classroom-based<br />
counterparts is their aut<strong>on</strong>omy (Keegan 1996) and greater student resp<strong>on</strong>sibility (Wedemeyer<br />
1981). By the time they reach higher educati<strong>on</strong>, most adults have acquired a degree <str<strong>on</strong>g>of</str<strong>on</strong>g> aut<strong>on</strong>omy<br />
in learning, but younger students need to be scaffolded as part <str<strong>on</strong>g>of</str<strong>on</strong>g> the distance educati<strong>on</strong><br />
experience. Virtual school teachers must be adept at helping children acquire the skills <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
aut<strong>on</strong>omous learning, including self-regulati<strong>on</strong>. Adult learners more closely approach expertise<br />
in the subjects they study, due to their l<strong>on</strong>g experience with the c<strong>on</strong>cepts, whereas children are<br />
novices. This distincti<strong>on</strong> is important because experts organize and interpret informati<strong>on</strong> very<br />
differently from novices, and these differences affect learners’ abilities to remember and solve<br />
problems (Bransford, Brown & Cocking 1999), and their ability to learn independently. Expert<br />
learners have better developed metacogniti<strong>on</strong>, a characteristic that children develop gradually.<br />
A sec<strong>on</strong>d characteristic that differentiates successful distance learners from unsuccessful <strong>on</strong>es is<br />
an internal locus <str<strong>on</strong>g>of</str<strong>on</strong>g> c<strong>on</strong>trol, leading them to persist in the educati<strong>on</strong>al endeavor (Rotter 1989).<br />
Research has found that older children have more internal locus <str<strong>on</strong>g>of</str<strong>on</strong>g> c<strong>on</strong>trol than younger children<br />
(Gershaw 1989), reinforcing the need for careful design and teaching <str<strong>on</strong>g>of</str<strong>on</strong>g> distance educati<strong>on</strong> at K-<br />
<strong>12</strong> levels.<br />
Young students are different from adult learners in other ways. Piaget’s stages <str<strong>on</strong>g>of</str<strong>on</strong>g> cognitive<br />
development, in particular preoperati<strong>on</strong>al (2 to 7 years), c<strong>on</strong>crete operati<strong>on</strong>al (7 to 11 years), and<br />
formal operati<strong>on</strong>al (11 years to adulthood), <str<strong>on</strong>g>of</str<strong>on</strong>g>fer pedagogical guidance for delivering effective<br />
web based educati<strong>on</strong>, focusing <strong>on</strong> the major accomplishments <str<strong>on</strong>g>of</str<strong>on</strong>g> learners in these stages. Each<br />
stage is characterized by the emergence <str<strong>on</strong>g>of</str<strong>on</strong>g> new abilities and ways <str<strong>on</strong>g>of</str<strong>on</strong>g> processing informati<strong>on</strong><br />
(Slavin 2003, p. 30), which necessitates specialized instructi<strong>on</strong>al approaches and attenti<strong>on</strong> to<br />
each child’s development. Since adults have progressed through these stages <str<strong>on</strong>g>of</str<strong>on</strong>g> cognitive<br />
development, delivery <str<strong>on</strong>g>of</str<strong>on</strong>g> web based educati<strong>on</strong> at the adult level need not c<strong>on</strong>centrate <strong>on</strong> methods<br />
that help the learner develop these cognitive skills. However, web based instructi<strong>on</strong> for students<br />
in their formative years must include age appropriate developmental activities, building <strong>on</strong> the<br />
students’ accomplishments in and through the cognitive stages.<br />
Piaget helps us to understand that learning should be holistic, authentic, and realistic. Less<br />
emphasis should be placed <strong>on</strong> isolated skills aimed at teaching individual c<strong>on</strong>cepts. <strong>Student</strong>s are<br />
more likely to learn skills while engaged in authentic, meaningful activities. Authentic activities<br />
4 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
are inherently interesting and meaningful to the student. Web-based technology <str<strong>on</strong>g>of</str<strong>on</strong>g>fers a vast<br />
array <str<strong>on</strong>g>of</str<strong>on</strong>g> opportunities to help expand the c<strong>on</strong>ceptual and experiential background <str<strong>on</strong>g>of</str<strong>on</strong>g> the student<br />
(Bolt<strong>on</strong> 2002, p. 5).<br />
Neo-Piagetian theorists have expanded <strong>on</strong> Piaget’s model <str<strong>on</strong>g>of</str<strong>on</strong>g> cognitive development. Am<strong>on</strong>g<br />
others, Vygotsky proposed that historical and cultural c<strong>on</strong>text play significant roles in helping<br />
people think, communicate, and solve problems, proposing that cognitive development is<br />
str<strong>on</strong>gly linked to input from others. Vygotsky’s theory implies that cognitive development and<br />
the ability to use thought to c<strong>on</strong>trol our own acti<strong>on</strong>s require first mastering cultural<br />
communicati<strong>on</strong> systems and then learning to use these systems to regulate our own thought<br />
process. He believed that learning takes place when children are working within their z<strong>on</strong>e <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
proximal development. Tasks within the z<strong>on</strong>e <str<strong>on</strong>g>of</str<strong>on</strong>g> proximal development are <strong>on</strong>es that children<br />
cannot yet do al<strong>on</strong>e but could do with the assistance <str<strong>on</strong>g>of</str<strong>on</strong>g> more competent peers or adults (Slavin<br />
2003, p. 43-44). When working with children using web-based technology, teachers must <str<strong>on</strong>g>of</str<strong>on</strong>g>fer<br />
students activities that make use <str<strong>on</strong>g>of</str<strong>on</strong>g> the web’s powerful tools for collaborative learning, and are<br />
within their z<strong>on</strong>e <str<strong>on</strong>g>of</str<strong>on</strong>g> proximal development. Online communities can provide a supportive<br />
c<strong>on</strong>text that makes new kinds <str<strong>on</strong>g>of</str<strong>on</strong>g> learning experiences possible (Bruckman 1998, p. 84-85).<br />
C<strong>on</strong>structivism, a widely used theory in distance educati<strong>on</strong>, is founded <strong>on</strong> the premise that by<br />
reflecting <strong>on</strong> our experiences we c<strong>on</strong>struct our understanding <str<strong>on</strong>g>of</str<strong>on</strong>g> the world we live in. Each <str<strong>on</strong>g>of</str<strong>on</strong>g> us<br />
generates our own "rules" and "mental models," which we use to make sense <str<strong>on</strong>g>of</str<strong>on</strong>g> our experiences.<br />
Learning, therefore, is simply the process <str<strong>on</strong>g>of</str<strong>on</strong>g> adjusting our mental models to accommodate new<br />
experiences (Brooks & Brooks 1993). Children have not had the experiences that adults have had<br />
to help them c<strong>on</strong>struct understanding. <str<strong>on</strong>g>The</str<strong>on</strong>g>refore, children c<strong>on</strong>struct an understanding <str<strong>on</strong>g>of</str<strong>on</strong>g> the<br />
world around them that lacks the rich experiences that adults have had. Scaffolding or mediated<br />
learning is important in helping children achieve these cognitive understandings (Slavin 2003, p.<br />
259), and are essential comp<strong>on</strong>ents <str<strong>on</strong>g>of</str<strong>on</strong>g> web-based learning experiences for children.<br />
With the emphasis <strong>on</strong> scientifically-based research and the call for evidence-based program<br />
decisi<strong>on</strong>s in the federal No Child Left Behind Act <str<strong>on</strong>g>of</str<strong>on</strong>g> 2001, detailed knowledge is needed to<br />
guide the growing numbers <str<strong>on</strong>g>of</str<strong>on</strong>g> cyber/<strong>on</strong>line/virtual school developers and educators. Metaanalysis<br />
is an established technique for synthesizing research findings to enable both a broader<br />
basis for understanding a phenomen<strong>on</strong> and a parsing <str<strong>on</strong>g>of</str<strong>on</strong>g> influences <strong>on</strong> the phenomen<strong>on</strong>. Several<br />
recent meta-analyses related to distance educati<strong>on</strong> have been published in recent years (Table 1).<br />
Table 1. Summary <str<strong>on</strong>g>of</str<strong>on</strong>g> recent meta-analyses in distance educati<strong>on</strong><br />
Author(s), Date Focus N <str<strong>on</strong>g>of</str<strong>on</strong>g> studies Effect Size<br />
Allen, Bourhis, <strong>Student</strong> satisfacti<strong>on</strong> 25 +0.031<br />
Burrell, Mabry, 2002. am<strong>on</strong>g adult learners<br />
Bernard, Abrami, <strong>Student</strong> achievement, 232 +0.0<strong>12</strong>8<br />
Lou, Borokhovski,<br />
Wade, Wozney,<br />
Wallet, Fiset, Huang,<br />
2003.<br />
attitude, retenti<strong>on</strong><br />
5 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
Cavanaugh, 2001. Academic<br />
achievement <str<strong>on</strong>g>of</str<strong>on</strong>g> K-<strong>12</strong><br />
students<br />
19 +0.015<br />
Machtmes, Asher,<br />
2000.<br />
Adult telecourses 30 -0.0093<br />
Shachar, Neumann, <strong>Student</strong> achievement 86 +0.37<br />
2003.<br />
Ungerleider, Burns,<br />
2003.<br />
Networked and <strong>on</strong>line<br />
learning<br />
6 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32<br />
<strong>12</strong> for achievement<br />
4 for satisfacti<strong>on</strong><br />
0 for achievement<br />
-0.509 for satisfacti<strong>on</strong><br />
Only <strong>on</strong>e <str<strong>on</strong>g>of</str<strong>on</strong>g> the recent meta-analyses in distance educati<strong>on</strong> focused <strong>on</strong> K-<strong>12</strong> learners, and it<br />
included web-based programs al<strong>on</strong>g with the analog c<strong>on</strong>ference and broadcast programs no<br />
l<strong>on</strong>ger in comm<strong>on</strong> use in today’s virtual schools. <str<strong>on</strong>g>The</str<strong>on</strong>g> explosi<strong>on</strong> in virtual schools, especially<br />
virtual charter schools in the United States, has necessitated a fresh look at the knowledge base.<br />
<str<strong>on</strong>g>The</str<strong>on</strong>g> need is for research that guides practiti<strong>on</strong>ers in refining practice so the most effective<br />
methods are used.<br />
Purpose <str<strong>on</strong>g>of</str<strong>on</strong>g> the Study<br />
A synthesis <str<strong>on</strong>g>of</str<strong>on</strong>g> studies <str<strong>on</strong>g>of</str<strong>on</strong>g> the effectiveness <str<strong>on</strong>g>of</str<strong>on</strong>g> distance educati<strong>on</strong> programs for K-<strong>12</strong> learners has a<br />
number <str<strong>on</strong>g>of</str<strong>on</strong>g> advantages. Because all <str<strong>on</strong>g>of</str<strong>on</strong>g> the studies included in this review drew data from schoolbased<br />
classes, the review can provide valuable insight into the practical effectiveness <str<strong>on</strong>g>of</str<strong>on</strong>g> K-<strong>12</strong><br />
distance educati<strong>on</strong>. C<strong>on</strong>trolled experimental research may <str<strong>on</strong>g>of</str<strong>on</strong>g>fer findings <str<strong>on</strong>g>of</str<strong>on</strong>g> theoretical interest<br />
but may not be generalizable to complex learning settings such as virtual schools or classes. <str<strong>on</strong>g>The</str<strong>on</strong>g><br />
unc<strong>on</strong>trollable cultural and social variables naturally present in a school or class, whether <strong>on</strong>line<br />
or <strong>on</strong>-ground, make a statistical synthesis a more exact test <str<strong>on</strong>g>of</str<strong>on</strong>g> the strength <str<strong>on</strong>g>of</str<strong>on</strong>g> K-<strong>12</strong> distance<br />
educati<strong>on</strong>. <str<strong>on</strong>g>The</str<strong>on</strong>g> effects <str<strong>on</strong>g>of</str<strong>on</strong>g> virtual learning would have to be str<strong>on</strong>g and c<strong>on</strong>sistent to be<br />
measurable across a range <str<strong>on</strong>g>of</str<strong>on</strong>g> natural milieus.<br />
<str<strong>on</strong>g>The</str<strong>on</strong>g> purpose <str<strong>on</strong>g>of</str<strong>on</strong>g> this meta-analysis is to provide a quantitative synthesis <str<strong>on</strong>g>of</str<strong>on</strong>g> the research literature<br />
<str<strong>on</strong>g>of</str<strong>on</strong>g> web-based K-<strong>12</strong> distance educati<strong>on</strong> from 1999 to the present, across c<strong>on</strong>tent areas, grade<br />
levels, and outcome measures. <str<strong>on</strong>g>The</str<strong>on</strong>g> first goal was to determine the effects <str<strong>on</strong>g>of</str<strong>on</strong>g> distance educati<strong>on</strong><br />
<strong>on</strong> K-<strong>12</strong> student outcomes, specifically academic achievement. <str<strong>on</strong>g>The</str<strong>on</strong>g> sec<strong>on</strong>d goal was to identify<br />
the effects <strong>on</strong> student outcomes <str<strong>on</strong>g>of</str<strong>on</strong>g> the features <str<strong>on</strong>g>of</str<strong>on</strong>g> distance educati<strong>on</strong>, including c<strong>on</strong>tent area,<br />
durati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> use, frequency <str<strong>on</strong>g>of</str<strong>on</strong>g> use, grade level <str<strong>on</strong>g>of</str<strong>on</strong>g> students, role <str<strong>on</strong>g>of</str<strong>on</strong>g> the instructor, type <str<strong>on</strong>g>of</str<strong>on</strong>g> school,<br />
timing <str<strong>on</strong>g>of</str<strong>on</strong>g> interacti<strong>on</strong>s, and pacing <str<strong>on</strong>g>of</str<strong>on</strong>g> the learning.<br />
From the literature, the meta-analysis seeks to answer the following questi<strong>on</strong>s:<br />
1. Is distance educati<strong>on</strong> as effective, in terms <str<strong>on</strong>g>of</str<strong>on</strong>g> student achievement, as classroom-based<br />
instructi<strong>on</strong>?<br />
2. To what extent are student outcomes related to the features <str<strong>on</strong>g>of</str<strong>on</strong>g> a distance educati<strong>on</strong> system<br />
(durati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> use, frequency <str<strong>on</strong>g>of</str<strong>on</strong>g> use, role <str<strong>on</strong>g>of</str<strong>on</strong>g> the instructor, timing <str<strong>on</strong>g>of</str<strong>on</strong>g> interacti<strong>on</strong>s, and pacing <str<strong>on</strong>g>of</str<strong>on</strong>g> the<br />
learning)?<br />
3. To what extent are student outcomes related to features <str<strong>on</strong>g>of</str<strong>on</strong>g> the educati<strong>on</strong>al c<strong>on</strong>text (c<strong>on</strong>tent<br />
area, school type, and grade level)?
4. To what extent are results related to study features (year, type <str<strong>on</strong>g>of</str<strong>on</strong>g> publicati<strong>on</strong>, various potential<br />
threats to validity)?<br />
Meta-analysis, the use <str<strong>on</strong>g>of</str<strong>on</strong>g> statistical analysis to synthesize a body <str<strong>on</strong>g>of</str<strong>on</strong>g> literature, is appropriate for<br />
answering questi<strong>on</strong>s such as these because it allows comparis<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> different studies by<br />
computing an effect size for each study. Meta-analysis is used to estimate the size <str<strong>on</strong>g>of</str<strong>on</strong>g> a<br />
treatment’s effect, and allows investigati<strong>on</strong> into relati<strong>on</strong>ships am<strong>on</strong>g study features and outcomes<br />
(Bangert-Drowns 2004). <str<strong>on</strong>g>The</str<strong>on</strong>g> inclusi<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> a study in a meta-analysis is limited by several factors,<br />
the most significant <str<strong>on</strong>g>of</str<strong>on</strong>g> which is the reporting <str<strong>on</strong>g>of</str<strong>on</strong>g> the informati<strong>on</strong> needed to compute effect size.<br />
Very <str<strong>on</strong>g>of</str<strong>on</strong>g>ten, reports released by virtual schools and other distance educati<strong>on</strong> programs do not<br />
include mean scores, comparis<strong>on</strong> group scores, sample sizes, or standard deviati<strong>on</strong>s.<br />
N<strong>on</strong>etheless, the meta-analytic technique is a way to identify effects or relati<strong>on</strong>ships in literature<br />
that may not be evident otherwise (Lipsey & Wils<strong>on</strong> 2001).<br />
Method<br />
This quantitative synthesis is a meta-analysis <str<strong>on</strong>g>of</str<strong>on</strong>g> empirical studies published since 1999 that<br />
compared the effects <str<strong>on</strong>g>of</str<strong>on</strong>g> web-delivered distance educati<strong>on</strong> with classroom-based learning <strong>on</strong> K-<br />
<strong>12</strong> student academic performance. Since 1999 the sophisticati<strong>on</strong> in the use <str<strong>on</strong>g>of</str<strong>on</strong>g> distance learning<br />
tools has improved, but the types <str<strong>on</strong>g>of</str<strong>on</strong>g> tools available to schools have remained approximately the<br />
same. <str<strong>on</strong>g>The</str<strong>on</strong>g> stages <str<strong>on</strong>g>of</str<strong>on</strong>g> the meta-analysis were identificati<strong>on</strong> and retrieval <str<strong>on</strong>g>of</str<strong>on</strong>g> applicable studies,<br />
coding <str<strong>on</strong>g>of</str<strong>on</strong>g> study features and findings, and data analysis. <str<strong>on</strong>g>The</str<strong>on</strong>g>se stages are described below.<br />
For the purposes <str<strong>on</strong>g>of</str<strong>on</strong>g> this meta-analysis, studies were included in the analysis if they met the<br />
following criteria for inclusi<strong>on</strong>. <str<strong>on</strong>g>The</str<strong>on</strong>g> studies must:<br />
• Be available as a journal article, dissertati<strong>on</strong> or report in English between 1999 and 2004.<br />
• Compare K-<strong>12</strong> students in a distance educati<strong>on</strong> group to a n<strong>on</strong>-distance educati<strong>on</strong> group,<br />
or compare the distance educati<strong>on</strong> group before and after distance educati<strong>on</strong>.<br />
• Use web-based telecommunicati<strong>on</strong>s, such that at least 50% <str<strong>on</strong>g>of</str<strong>on</strong>g> the students’ participati<strong>on</strong><br />
in the course or program occurred at a physical distance from the instructor.<br />
• Be quantitative, experimental, and quasi-experimental studies for which effect size could<br />
be computed, the outcome measures were the same or comparable, and the N was 2 or<br />
greater.<br />
• Use student academic achievement, motivati<strong>on</strong>, attitude, retenti<strong>on</strong>, or behavior as<br />
outcome variables.<br />
Locati<strong>on</strong> and selecti<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> studies<br />
Numerous databases, journals, websites, and bibliographic resources were searched for studies<br />
that met the established inclusi<strong>on</strong> criteria. In each case, search terms included:<br />
cybercharter<br />
cyberschool<br />
distance educati<strong>on</strong><br />
distance learning<br />
7 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
elearning<br />
mlearning<br />
<strong>on</strong>line school<br />
open learning<br />
open school<br />
schoolnet<br />
telelearning<br />
virtual charter<br />
virtual school.<br />
Electr<strong>on</strong>ic searches were systematically c<strong>on</strong>ducted in the following databases:<br />
Dissertati<strong>on</strong> Abstracts<br />
ERIC<br />
JSTOR<br />
Kluwer<br />
ProQuest <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g><br />
PsychInfo<br />
Wils<strong>on</strong> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>.<br />
Web searches were performed using the Google, Teoma, Grokker, MetaCrawler, and AltaVista<br />
search sites.<br />
Abstracts in the following distance educati<strong>on</strong> journals were examined:<br />
American Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Distance</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g><br />
Computers & <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g><br />
<str<strong>on</strong>g>Distance</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g><br />
Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Distance</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g><br />
Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Distance</str<strong>on</strong>g> Learning<br />
Open Learning.<br />
Abstracts in the following educati<strong>on</strong>al technology journals were examined:<br />
Associati<strong>on</strong> for the Advancement <str<strong>on</strong>g>of</str<strong>on</strong>g> Computing in <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g> journals<br />
Australasian Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>al Technology<br />
British Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>al Technology<br />
Canadian Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>al Communicati<strong>on</strong><br />
Canadian Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> Learning and Technology<br />
Computers in the School<br />
<str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>al Technology & Society<br />
<str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>al Technology Research and Development<br />
Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> Computer Mediated Communicati<strong>on</strong><br />
Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> Computing in Childhood <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g><br />
Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>al Computing Research<br />
Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> Informati<strong>on</strong> Technology <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g><br />
Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> Interactive Media in <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g><br />
Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> Research <strong>on</strong> Technology in <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>.<br />
8 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
Abstracts in American <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>al Research Journal were examined, as were abstracts in the<br />
following electr<strong>on</strong>ic journals:<br />
Australian <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>al Computing<br />
Australian Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>al Technology<br />
Electr<strong>on</strong>ic Journal for the Integrati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> Technology in <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g><br />
Internati<strong>on</strong>al Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>al Technology<br />
Internati<strong>on</strong>al Review <str<strong>on</strong>g>of</str<strong>on</strong>g> Research in Open and <str<strong>on</strong>g>Distance</str<strong>on</strong>g> Learning<br />
Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> Asynchr<strong>on</strong>ous Learning Networks<br />
Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> Interactive Online Learning<br />
Online Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Distance</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g> Administrati<strong>on</strong><br />
TechKnowLogia: Internati<strong>on</strong>al Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> Technologies for the Advancement <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
Knowledge and Learning<br />
Turkish Online Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Distance</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>.<br />
In additi<strong>on</strong>, abstracts were examined in the following c<strong>on</strong>ference proceedings:<br />
American <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g> Research Associati<strong>on</strong><br />
Canadian Associati<strong>on</strong> for <str<strong>on</strong>g>Distance</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g><br />
EdMedia<br />
E-Learn/WebNet<br />
Society for Technology in Teacher <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>.<br />
<str<strong>on</strong>g>The</str<strong>on</strong>g> web sites <str<strong>on</strong>g>of</str<strong>on</strong>g> several distance educati<strong>on</strong> organizati<strong>on</strong>s and over 200 virtual schools were<br />
browsed for studies, and the director <str<strong>on</strong>g>of</str<strong>on</strong>g> each virtual school was c<strong>on</strong>tacted at the email address<br />
listed <strong>on</strong> the school’s website to request studies. <str<strong>on</strong>g>The</str<strong>on</strong>g> department <str<strong>on</strong>g>of</str<strong>on</strong>g> educati<strong>on</strong> website for each<br />
state was browsed for report cards for state virtual charter schools. <str<strong>on</strong>g>The</str<strong>on</strong>g> reference lists <str<strong>on</strong>g>of</str<strong>on</strong>g> the six<br />
recent meta-analyses <str<strong>on</strong>g>of</str<strong>on</strong>g> distance educati<strong>on</strong> shown in Table 1 were reviewed for potential studies.<br />
Of the thousands <str<strong>on</strong>g>of</str<strong>on</strong>g> abstracts that were reviewed, 80 full-text articles, dissertati<strong>on</strong>s, or reports<br />
c<strong>on</strong>cerning DE and traditi<strong>on</strong>al instructi<strong>on</strong> at K-<strong>12</strong> level were obtained and evaluated for<br />
inclusi<strong>on</strong> in the analysis. Independently, two researchers read all collected studies to determine<br />
eligibility for inclusi<strong>on</strong> based <strong>on</strong> the stated criteria. Fourteen <str<strong>on</strong>g>of</str<strong>on</strong>g> the studies were found to meet<br />
all criteria for inclusi<strong>on</strong>. Of the 66 studies that were examined and excluded, 28% were<br />
descriptive reports, 14% reported <strong>on</strong> uses <str<strong>on</strong>g>of</str<strong>on</strong>g> telecommunicati<strong>on</strong>s or other educati<strong>on</strong>al technology<br />
that did not meet the definiti<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> distance educati<strong>on</strong>, 25% reported results without c<strong>on</strong>trol or<br />
comparis<strong>on</strong> group data, and 33% included summary data <strong>on</strong>ly or did not provide data sufficient<br />
to compute effect size.<br />
Limitati<strong>on</strong>s <str<strong>on</strong>g>of</str<strong>on</strong>g> the Review<br />
For literature <strong>on</strong> K-<strong>12</strong> distance educati<strong>on</strong> to be meaningfully synthesized, the inclusi<strong>on</strong> criteria<br />
had to be narrowly specified. This synthesis included studies with data <strong>on</strong> the performance <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
grades 3-<strong>12</strong> students in web-based distance learning programs to students in classrooms.<br />
Measures <str<strong>on</strong>g>of</str<strong>on</strong>g> performance present in the literature do not draw a complete picture <str<strong>on</strong>g>of</str<strong>on</strong>g> the full<br />
range <str<strong>on</strong>g>of</str<strong>on</strong>g> effects that students experience as a result <str<strong>on</strong>g>of</str<strong>on</strong>g> participati<strong>on</strong> in distance educati<strong>on</strong>.<br />
Qualitative studies, strict experimental studies, narrative reports, and other designs <str<strong>on</strong>g>of</str<strong>on</strong>g>fer<br />
9 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
informati<strong>on</strong> not acquired in this analysis. Although the inclusi<strong>on</strong> criteria were designed to allow<br />
a wide range <str<strong>on</strong>g>of</str<strong>on</strong>g> studies to be analyzed so that a comprehensive knowledge <str<strong>on</strong>g>of</str<strong>on</strong>g> K-<strong>12</strong> distance<br />
educati<strong>on</strong> would result, a small number <str<strong>on</strong>g>of</str<strong>on</strong>g> studies was analyzed. <str<strong>on</strong>g>The</str<strong>on</strong>g> results should be<br />
interpreted with cauti<strong>on</strong>.<br />
Coding <str<strong>on</strong>g>of</str<strong>on</strong>g> study features<br />
Coding <str<strong>on</strong>g>of</str<strong>on</strong>g> study features allows the meta-analyst to unravel different study factors related to<br />
variati<strong>on</strong>s in the phenomen<strong>on</strong> from factors related to method (Lipsey & Wils<strong>on</strong> 2001). <str<strong>on</strong>g>The</str<strong>on</strong>g><br />
coding used in this analysis was identified from research <strong>on</strong> K-<strong>12</strong> distance educati<strong>on</strong> and from<br />
variables typically coded in c<strong>on</strong>temporary meta-analyses in educati<strong>on</strong>. A trial c<strong>on</strong>ducted <strong>on</strong> a<br />
small sample <str<strong>on</strong>g>of</str<strong>on</strong>g> studies led to the additi<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> variables in the codebook that were not present in<br />
the initial set <str<strong>on</strong>g>of</str<strong>on</strong>g> variables. Each study was coded independently by two researchers according to<br />
the established coding procedure. <str<strong>on</strong>g>The</str<strong>on</strong>g> full codebook is included in Appendix A. <str<strong>on</strong>g>The</str<strong>on</strong>g> initial interrater<br />
agreement across all coded variables was 85%. Discrepancies between researchers were<br />
discussed and resolved. <str<strong>on</strong>g>The</str<strong>on</strong>g> entire dataset was reviewed for the presence <str<strong>on</strong>g>of</str<strong>on</strong>g> discrepancies and<br />
unexpected values.<br />
Fourteen studies, with a total <str<strong>on</strong>g>of</str<strong>on</strong>g> 116 outcomes, had data sufficient to include in the analysis. <str<strong>on</strong>g>The</str<strong>on</strong>g><br />
dependent variable in this synthesis was student outcome measured by instruments appropriate to<br />
the individual study given at the end <str<strong>on</strong>g>of</str<strong>on</strong>g> the distance educati<strong>on</strong> period which varied from a few<br />
weeks to an entire academic year. <str<strong>on</strong>g>The</str<strong>on</strong>g> measures included district, state, or nati<strong>on</strong>al examinati<strong>on</strong>s;<br />
and teacher or researcher designed tests <str<strong>on</strong>g>of</str<strong>on</strong>g> academic performance.<br />
<str<strong>on</strong>g>The</str<strong>on</strong>g> studies were coded <strong>on</strong> 45 factors, categorized into five groups: identificati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> studies,<br />
distance educati<strong>on</strong> features, instructor/program features, study quality features, and sources <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
invalidity. Of particular interest were the variables associated with distance educati<strong>on</strong> features<br />
(e.g. durati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> the experience, role <str<strong>on</strong>g>of</str<strong>on</strong>g> the distance learning, role <str<strong>on</strong>g>of</str<strong>on</strong>g> the instructor, timing <str<strong>on</strong>g>of</str<strong>on</strong>g> the<br />
interacti<strong>on</strong>s) and instructor/program features (e.g. amount <str<strong>on</strong>g>of</str<strong>on</strong>g> teacher preparati<strong>on</strong> for distance<br />
teaching, setting <str<strong>on</strong>g>of</str<strong>on</strong>g> the students). In many cases, however, the literature failed to report the detail<br />
needed to make meaningful comparis<strong>on</strong>s <strong>on</strong> these factors. <str<strong>on</strong>g>The</str<strong>on</strong>g> levels <str<strong>on</strong>g>of</str<strong>on</strong>g> each variable were<br />
compared by computing average effect sizes for each level, but examinati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> interacti<strong>on</strong>s<br />
am<strong>on</strong>g the different variables was not practical due to the small number <str<strong>on</strong>g>of</str<strong>on</strong>g> effect sizes available.<br />
Calculati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> effect sizes<br />
<str<strong>on</strong>g>The</str<strong>on</strong>g> effect sizes estimated for each study outcome were computed using Cohen’s d, defined in<br />
this meta-analysis as the difference between the n<strong>on</strong>-distance learning group and the distance<br />
learning posttest mean scores divided by the average standard deviati<strong>on</strong>. A correcti<strong>on</strong> factor for<br />
small sample bias in effect size estimati<strong>on</strong> (Hedges, Shymansky & Woodworth 1989) was used<br />
in cases in which sample sizes were small. <str<strong>on</strong>g>The</str<strong>on</strong>g> unit <str<strong>on</strong>g>of</str<strong>on</strong>g> analysis was the study outcome. For<br />
studies in which more than <strong>on</strong>e independent group <str<strong>on</strong>g>of</str<strong>on</strong>g> students was evaluated, independent effect<br />
sizes were estimated for each group, were weighted to avoid study bias, and were included in the<br />
aggregated effect size estimate. A positive effect size, with a 95% c<strong>on</strong>fidence interval not<br />
10 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
encompassing zero, is an indicati<strong>on</strong> that the distance learning group outperformed the n<strong>on</strong>distance<br />
learning group.<br />
Table 2. Selected study features and effect sizes for 14 studies <str<strong>on</strong>g>of</str<strong>on</strong>g> web-based K-<strong>12</strong> distance<br />
educati<strong>on</strong><br />
Author, year Grade<br />
level<br />
Alberta<br />
C<strong>on</strong>sortium<br />
2001*<br />
Alaska<br />
Department <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
<str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g> and<br />
Early<br />
Development<br />
2003*<br />
Colorado<br />
Department <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
<str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g><br />
2003a*<br />
Colorado<br />
Department <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
<str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g><br />
2003b*<br />
Colorado<br />
Department <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
<str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g><br />
2003c*<br />
Colorado<br />
Department <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
<str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g><br />
2003d*<br />
Goc Karp &<br />
Woods 2003*<br />
Indiana<br />
Department <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
<str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>,<br />
2004*<br />
Minnesota<br />
Department <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
<str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g><br />
2003*<br />
3, 6, 9,<br />
<strong>12</strong><br />
4-7, 9-<br />
<strong>12</strong><br />
Subject area School<br />
type<br />
English,<br />
Mathematics,<br />
Science,<br />
Social Studies<br />
Reading,<br />
Writing,<br />
Mathematics<br />
3-6 Reading,<br />
Writing,<br />
Mathematics<br />
7-8 Reading,<br />
Writing,<br />
Mathematics<br />
3-6 Reading,<br />
Writing,<br />
Mathematics<br />
7-8 Reading,<br />
Writing,<br />
Mathematics<br />
9-<strong>12</strong> Physical<br />
educati<strong>on</strong><br />
3, 6 Reading,<br />
mathematics<br />
5 Reading,<br />
mathematics<br />
Mix <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
public<br />
and<br />
private<br />
State<br />
charter<br />
State<br />
charter<br />
State<br />
charter<br />
State<br />
charter<br />
State<br />
charter<br />
Outcome<br />
measure<br />
Nati<strong>on</strong>al<br />
test<br />
State and<br />
nati<strong>on</strong>al<br />
tests<br />
11 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32<br />
Instructi<strong>on</strong>al<br />
role <str<strong>on</strong>g>of</str<strong>on</strong>g> the<br />
distance<br />
learning<br />
Timing <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
interacti<strong>on</strong>s<br />
N Weighted<br />
mean<br />
effect size<br />
(d)<br />
95% CI<br />
for d<br />
(upper/lo<br />
wer)<br />
Course Asynchr<strong>on</strong>ous 13-397 -0.028 0.141/-<br />
0.197<br />
Full program Synchr<strong>on</strong>ous 7-67 -0.005 0.303/-<br />
0.313<br />
State test Full program Asynchr<strong>on</strong>ous 33-45 -0.028 0.261/-<br />
0.276<br />
State test Full program Asynchr<strong>on</strong>ous 9-55 -0.029 0.199/-<br />
0.258<br />
State test Full program Combinati<strong>on</strong><br />
synchr<strong>on</strong>ous<br />
asynchr<strong>on</strong>ous<br />
State test Full program Combinati<strong>on</strong><br />
synchr<strong>on</strong>ous<br />
asynchr<strong>on</strong>ous<br />
14-23 -0.013 0.440/-<br />
0.466<br />
10-21 -0.013 0.449/-<br />
0.475<br />
Public Class Porti<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> Asynchr<strong>on</strong>ous 19 -0.253 0.357/assignments<br />
course<br />
0.863<br />
State State test Full program Unspecified 17-18 0.001 0.470/charter<br />
0.468<br />
State<br />
charter<br />
State test Full program Unspecified 26 0.014 0.398/-<br />
0.371<br />
Mock 2000* <strong>12</strong> Science Public Teacher Porti<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> Asynchr<strong>on</strong>ous 7 -0.472 0.472/made<br />
test course<br />
1.416<br />
Stevens 1999* <strong>12</strong> Science Public Teacher Porti<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> Unspecified 21-33 -0.029 0.497/made<br />
test course<br />
0.556<br />
Washingt<strong>on</strong> 7 Reading, State State test Full program Asynchr<strong>on</strong>ous <strong>12</strong>-15 0.002 0.540/-<br />
Office <str<strong>on</strong>g>of</str<strong>on</strong>g> the<br />
Writing, charter<br />
0.537<br />
Superintendent<br />
Mathematics,<br />
<str<strong>on</strong>g>of</str<strong>on</strong>g> Public<br />
Instructi<strong>on</strong><br />
2003*<br />
Listening<br />
Wisc<strong>on</strong>sin 3 Reading State State test Full program Asynchr<strong>on</strong>ous 57 -0.016 0.243/-<br />
Department <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
Public<br />
charter<br />
0.276
Instructi<strong>on</strong><br />
2003<br />
Texas<br />
<str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g><br />
Agency 2003*<br />
9-11 English,<br />
Mathematics,<br />
Science,<br />
Social Studies<br />
State<br />
charter<br />
* indicates studies yielding multiple effect sizes<br />
State test Full program Combinati<strong>on</strong> 15-21 -0.014 0.445/-<br />
0.474<br />
Statistical analysis <str<strong>on</strong>g>of</str<strong>on</strong>g> effect sizes<br />
<str<strong>on</strong>g>The</str<strong>on</strong>g> test for heterogeneity (Q), based <strong>on</strong> Hedges and Olkin (1985), was used to determine<br />
whether the effect sizes <str<strong>on</strong>g>of</str<strong>on</strong>g> the studies were homogenously distributed, in other words, to learn<br />
whether the distributi<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> effect sizes around their mean was what would be expected from<br />
sampling error al<strong>on</strong>e (Lipsey & Wils<strong>on</strong> 2001). <str<strong>on</strong>g>The</str<strong>on</strong>g> Q value for the weighted effect sizes was<br />
1.485, and was c<strong>on</strong>sidered to be homogeneous, indicating that the variance observed was likely<br />
to be due to sampling error. <str<strong>on</strong>g>The</str<strong>on</strong>g>refore, the fixed-effects model was used to estimate variance<br />
(Kromrey & Hogarty 2002). Study feature analyses were performed to determine the extent to<br />
which student outcomes were moderated by the study variables. Statistical Analysis System<br />
(SAS) s<str<strong>on</strong>g>of</str<strong>on</strong>g>tware was used for the analyses. Effect size comparis<strong>on</strong>s were d<strong>on</strong>e for the variables:<br />
grade level, c<strong>on</strong>tent area, durati<strong>on</strong> and frequency <str<strong>on</strong>g>of</str<strong>on</strong>g> the distance learning experience,<br />
instructi<strong>on</strong>al role <str<strong>on</strong>g>of</str<strong>on</strong>g> the distance educati<strong>on</strong>, pacing <str<strong>on</strong>g>of</str<strong>on</strong>g> the instructi<strong>on</strong>, role <str<strong>on</strong>g>of</str<strong>on</strong>g> the instructor,<br />
timing <str<strong>on</strong>g>of</str<strong>on</strong>g> the interacti<strong>on</strong>s, and types <str<strong>on</strong>g>of</str<strong>on</strong>g> interacti<strong>on</strong>s, as well as for various study quality and<br />
invalidity factors.<br />
Results<br />
Characteristics <str<strong>on</strong>g>of</str<strong>on</strong>g> the Studies<br />
<str<strong>on</strong>g>The</str<strong>on</strong>g> 14 studies included in the analysis yielded 116 independent effect sizes drawn from a<br />
combined sample <str<strong>on</strong>g>of</str<strong>on</strong>g> 7561 students whose performance as a result <str<strong>on</strong>g>of</str<strong>on</strong>g> participati<strong>on</strong> in a distance<br />
educati<strong>on</strong> program was compared to c<strong>on</strong>trol groups in which students did not participate in<br />
distance educati<strong>on</strong>. Sixty <strong>on</strong>e percent <str<strong>on</strong>g>of</str<strong>on</strong>g> the study results had sample sizes <str<strong>on</strong>g>of</str<strong>on</strong>g> less than 50, and<br />
16% had sample sizes above 100. All but <strong>on</strong>e <str<strong>on</strong>g>of</str<strong>on</strong>g> the studies included more than <strong>on</strong>e comparis<strong>on</strong>,<br />
and the average number <str<strong>on</strong>g>of</str<strong>on</strong>g> comparis<strong>on</strong>s per study was 8, ranging from <strong>on</strong>e to 38. Eighty six<br />
percent <str<strong>on</strong>g>of</str<strong>on</strong>g> the studies were organizati<strong>on</strong> reports, 7% were published articles, and 7% were<br />
dissertati<strong>on</strong>s. All <str<strong>on</strong>g>of</str<strong>on</strong>g> the studies were published between 1999 and 2004, with eleven published<br />
during 2003 and 2004, and three published from 1999 through 2001. Two studies were published<br />
in Canada, and the other twelve were published in the U.S.<br />
A range <str<strong>on</strong>g>of</str<strong>on</strong>g> distance learning structures was examined in the literature. Half <str<strong>on</strong>g>of</str<strong>on</strong>g> the studies<br />
reported <strong>on</strong> programs that used asynchr<strong>on</strong>ous timing in instructi<strong>on</strong>. Three studies documented a<br />
program that used a combinati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> synchr<strong>on</strong>ous and asynchr<strong>on</strong>ous instructi<strong>on</strong>, <strong>on</strong>e program was<br />
delivered synchr<strong>on</strong>ously, and the remaining programs did not report <strong>on</strong> instructi<strong>on</strong>al timing. Ten<br />
<str<strong>on</strong>g>of</str<strong>on</strong>g> the studies reported results <str<strong>on</strong>g>of</str<strong>on</strong>g> student participati<strong>on</strong> in full year-l<strong>on</strong>g distance learning<br />
programs, <strong>on</strong>e included data for distance learning courses, and three studies focused <strong>on</strong> porti<strong>on</strong>s<br />
<str<strong>on</strong>g>of</str<strong>on</strong>g> courses delivered at a distance for less than a semester. Thirteen studies included data from<br />
<strong>12</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
programs in which students participated approximately five days per week, and the other study<br />
did not indicate the frequency <str<strong>on</strong>g>of</str<strong>on</strong>g> student participati<strong>on</strong>.<br />
<str<strong>on</strong>g>The</str<strong>on</strong>g> studies encompassed a variety <str<strong>on</strong>g>of</str<strong>on</strong>g> instructi<strong>on</strong>al features. <str<strong>on</strong>g>The</str<strong>on</strong>g> bulk <str<strong>on</strong>g>of</str<strong>on</strong>g> the results, 75%,<br />
occurred in the sec<strong>on</strong>dary grades, 6-<strong>12</strong>. <str<strong>on</strong>g>The</str<strong>on</strong>g> other results c<strong>on</strong>cern elementary age children, in<br />
grades 3-5. Results from seven academic c<strong>on</strong>tent areas were reported. Thirty percent <str<strong>on</strong>g>of</str<strong>on</strong>g> the<br />
results came from tests <str<strong>on</strong>g>of</str<strong>on</strong>g> reading ability, followed by mathematics, which accounted for 26% <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
the results. Writing was the subject for 16% <str<strong>on</strong>g>of</str<strong>on</strong>g> the results, science was the topic <str<strong>on</strong>g>of</str<strong>on</strong>g> 14%, and<br />
social studies made up 9% <str<strong>on</strong>g>of</str<strong>on</strong>g> the results. Three percent <str<strong>on</strong>g>of</str<strong>on</strong>g> results came from physical educati<strong>on</strong><br />
comparis<strong>on</strong>s, and <strong>on</strong>e percent from a test <str<strong>on</strong>g>of</str<strong>on</strong>g> listening. Nati<strong>on</strong>al tests were used to compare<br />
outcomes in <strong>on</strong>e study, state tests were used in nine studies, teacher made tests were used in two<br />
studies, and <strong>on</strong>e study reported data from both state and nati<strong>on</strong>al tests.<br />
Overall <str<strong>on</strong>g>Effects</str<strong>on</strong>g> <str<strong>on</strong>g>of</str<strong>on</strong>g> K-<strong>12</strong> <str<strong>on</strong>g>Distance</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g><br />
<str<strong>on</strong>g>The</str<strong>on</strong>g> analysis resulted in an overall weighted effect size not significantly different from zero, a<br />
result that is in line with the results <str<strong>on</strong>g>of</str<strong>on</strong>g> recent meta-analyses <str<strong>on</strong>g>of</str<strong>on</strong>g> distance educati<strong>on</strong> (see Table 1),<br />
which tend to show that distance educati<strong>on</strong> is as effective as classroom instructi<strong>on</strong>. <str<strong>on</strong>g>The</str<strong>on</strong>g> weighted<br />
mean effect size across all results was -0.028, with a standard error <str<strong>on</strong>g>of</str<strong>on</strong>g> 0.045 and a 95%<br />
c<strong>on</strong>fidence interval from -0.116 to 0.060. <str<strong>on</strong>g>The</str<strong>on</strong>g> average unweighted Cohen’s d was -0.034, and the<br />
median effect size was -0.015. <str<strong>on</strong>g>The</str<strong>on</strong>g> effect sizes varied c<strong>on</strong>siderably am<strong>on</strong>g the studies (see Figure<br />
1). Unweighted effect sizes ranged from -1.158 to 0.597, with a standard deviati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> 0.157,<br />
indicating that some applicati<strong>on</strong>s <str<strong>on</strong>g>of</str<strong>on</strong>g> distance educati<strong>on</strong> appeared to be much better than<br />
classroom instructi<strong>on</strong> and others were much worse. However, each <str<strong>on</strong>g>of</str<strong>on</strong>g> the fourteen studies and all<br />
except <strong>on</strong>e <str<strong>on</strong>g>of</str<strong>on</strong>g> the 116 outcomes within the studies had individual effect sizes that did not differ<br />
significantly from zero.<br />
Figure 1. Distributi<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> unweighted effect sizes<br />
Number <str<strong>on</strong>g>of</str<strong>on</strong>g> results<br />
Distributi<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> unweighted effect sizes<br />
-1.4 -1.2 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8<br />
Effect size<br />
13 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32<br />
100<br />
80<br />
60<br />
40<br />
20<br />
0
Figure 2. C<strong>on</strong>fidence intervals for individual effect sizes <str<strong>on</strong>g>of</str<strong>on</strong>g> 116 outcomes<br />
1.5<br />
1.0<br />
0.5<br />
0.0<br />
-0.5<br />
-1.0<br />
-1.5<br />
-2.0<br />
C<strong>on</strong>fidence Intervals for Individual Effect Sizes<br />
Of the 45 factors coded in the study, the following 30 were examined to determine sources <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
significant variati<strong>on</strong> in effect sizes. Ten <str<strong>on</strong>g>of</str<strong>on</strong>g> the remaining variables were used for identifying the<br />
studies or computing effect size, and the other five could not be compared because the studies<br />
did not include the data for coding the variables, or the variable was not a relevant factor in the<br />
studies. <str<strong>on</strong>g>The</str<strong>on</strong>g> variables that went uncoded due to the absence <str<strong>on</strong>g>of</str<strong>on</strong>g> data were the frequency <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
student participati<strong>on</strong> in distance learning, the level <str<strong>on</strong>g>of</str<strong>on</strong>g> preparati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> the teachers in distance<br />
educati<strong>on</strong>, and the amount <str<strong>on</strong>g>of</str<strong>on</strong>g> experience <str<strong>on</strong>g>of</str<strong>on</strong>g> the teachers in distance educati<strong>on</strong>. <str<strong>on</strong>g>The</str<strong>on</strong>g> variables that<br />
were not relevant factors for the studies were c<strong>on</strong>trol for the effects <str<strong>on</strong>g>of</str<strong>on</strong>g> a sec<strong>on</strong>d testing, and<br />
c<strong>on</strong>trol for the effects <str<strong>on</strong>g>of</str<strong>on</strong>g> a pretest. Analysis <str<strong>on</strong>g>of</str<strong>on</strong>g> variance was not meaningful for some <str<strong>on</strong>g>of</str<strong>on</strong>g> the<br />
variables because <str<strong>on</strong>g>of</str<strong>on</strong>g> missing data in the studies, resulting in a high number <str<strong>on</strong>g>of</str<strong>on</strong>g> cases in which a<br />
value <str<strong>on</strong>g>of</str<strong>on</strong>g> “unspecified” was coded for the variable.<br />
Publicati<strong>on</strong> and Methodological Variables<br />
Twenty variables were coded to discover whether publicati<strong>on</strong> or methodological variables<br />
accounted for variati<strong>on</strong> in effect sizes. <str<strong>on</strong>g>The</str<strong>on</strong>g> publicati<strong>on</strong> features included the year <str<strong>on</strong>g>of</str<strong>on</strong>g> publicati<strong>on</strong>,<br />
the type <str<strong>on</strong>g>of</str<strong>on</strong>g> publicati<strong>on</strong>, and the regi<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> publicati<strong>on</strong>. <str<strong>on</strong>g>The</str<strong>on</strong>g> methodological variables related to the<br />
testing sequence in the study, the type <str<strong>on</strong>g>of</str<strong>on</strong>g> achievement measure used in achievement studies,<br />
pretest equivalency measures, study design, statistical power, and c<strong>on</strong>trol for <strong>12</strong> potential sources<br />
14 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
<str<strong>on</strong>g>of</str<strong>on</strong>g> invalidity. N<strong>on</strong>e <str<strong>on</strong>g>of</str<strong>on</strong>g> the variable comparis<strong>on</strong>s resulted in effect sizes significantly different<br />
from zero.<br />
<str<strong>on</strong>g>Distance</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g> Variables<br />
Eleven variables were used to identify the features <str<strong>on</strong>g>of</str<strong>on</strong>g> the distance educati<strong>on</strong> experience that may<br />
play a role in student performance. <str<strong>on</strong>g>The</str<strong>on</strong>g>y were durati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> the program, frequency <str<strong>on</strong>g>of</str<strong>on</strong>g> use <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
distance learning, instructi<strong>on</strong>al role <str<strong>on</strong>g>of</str<strong>on</strong>g> the program, number <str<strong>on</strong>g>of</str<strong>on</strong>g> distance learning sessi<strong>on</strong>s,<br />
durati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> distance learning sessi<strong>on</strong>s, pacing <str<strong>on</strong>g>of</str<strong>on</strong>g> the instructi<strong>on</strong>, role <str<strong>on</strong>g>of</str<strong>on</strong>g> the instructor, timing <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
the interacti<strong>on</strong>s, type <str<strong>on</strong>g>of</str<strong>on</strong>g> interacti<strong>on</strong>s, amount <str<strong>on</strong>g>of</str<strong>on</strong>g> teacher preparati<strong>on</strong> for distance instructi<strong>on</strong>, and<br />
amount <str<strong>on</strong>g>of</str<strong>on</strong>g> teacher experience in distance instructi<strong>on</strong>. Because <str<strong>on</strong>g>of</str<strong>on</strong>g> the individualized nature <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
distance educati<strong>on</strong>, <strong>on</strong>ly two <str<strong>on</strong>g>of</str<strong>on</strong>g> the studies indicated specific numbers and durati<strong>on</strong>s <str<strong>on</strong>g>of</str<strong>on</strong>g> distance<br />
learning sessi<strong>on</strong>s, and they were studies <str<strong>on</strong>g>of</str<strong>on</strong>g> limited partial-course experiences. Half <str<strong>on</strong>g>of</str<strong>on</strong>g> the studies<br />
did not indicate whether students or instructors set the pace within the distance learning<br />
timeframe, while three <str<strong>on</strong>g>of</str<strong>on</strong>g> the programs were completely self-paced, and four were designed for<br />
students to set their pace within parameters set by the instructor. In terms <str<strong>on</strong>g>of</str<strong>on</strong>g> the role <str<strong>on</strong>g>of</str<strong>on</strong>g> the<br />
instructor in teaching, <strong>on</strong>e program was fully moderated, five were n<strong>on</strong>-moderated, four used a<br />
combinati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> moderated and n<strong>on</strong>-moderated activities, and four did not indicate the<br />
instructors’ role. Ten programs used a combinati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> interacti<strong>on</strong>s am<strong>on</strong>g students, c<strong>on</strong>tent,<br />
instructors, and others; <strong>on</strong>e limited interacti<strong>on</strong>s to student-c<strong>on</strong>tent; and three did not specify<br />
interacti<strong>on</strong> types. No studies described the levels <str<strong>on</strong>g>of</str<strong>on</strong>g> instructor preparati<strong>on</strong> or experience required<br />
<str<strong>on</strong>g>of</str<strong>on</strong>g> or possessed by the instructors. All levels <str<strong>on</strong>g>of</str<strong>on</strong>g> each distance educati<strong>on</strong> variable had effect sizes<br />
not significantly different from zero.<br />
Instructi<strong>on</strong>al and Program Variables<br />
<str<strong>on</strong>g>The</str<strong>on</strong>g> five variables that indicated the extent to which instructi<strong>on</strong>al and program factors played a<br />
role in student outcomes were grade level, school type, c<strong>on</strong>tent area, the qualificati<strong>on</strong>s <str<strong>on</strong>g>of</str<strong>on</strong>g> the<br />
teacher in the teaching field, and the setting <str<strong>on</strong>g>of</str<strong>on</strong>g> the students. Twelve <str<strong>on</strong>g>of</str<strong>on</strong>g> the studies indicated that<br />
the instructors were certified teachers, and the other two studies did not describe the credentials<br />
<str<strong>on</strong>g>of</str<strong>on</strong>g> the instructors. In five <str<strong>on</strong>g>of</str<strong>on</strong>g> the programs, students participated from home or a n<strong>on</strong>-school<br />
locati<strong>on</strong>, four programs are designed such that students completed some work from home and<br />
some in a school setting, in three programs, students did their distance learning work while at a<br />
school, and two programs did not specify the setting <str<strong>on</strong>g>of</str<strong>on</strong>g> the students. All instructi<strong>on</strong>al and<br />
program factors had effect sizes that were effectively zero.<br />
Discussi<strong>on</strong><br />
<str<strong>on</strong>g>The</str<strong>on</strong>g> literature reviewed in this meta-analysis includes results from 116 comparis<strong>on</strong>s <str<strong>on</strong>g>of</str<strong>on</strong>g> grades 3-<br />
<strong>12</strong> web-based distance educati<strong>on</strong> programs with classroom-based teaching, including data for<br />
7561 students. <str<strong>on</strong>g>The</str<strong>on</strong>g> questi<strong>on</strong>s <str<strong>on</strong>g>of</str<strong>on</strong>g> the effectiveness <str<strong>on</strong>g>of</str<strong>on</strong>g> distance educati<strong>on</strong> for K-<strong>12</strong> student<br />
performance, and <str<strong>on</strong>g>of</str<strong>on</strong>g> the factors influencing its effectiveness were addressed using fixed-effects<br />
effect size estimati<strong>on</strong>. <str<strong>on</strong>g>The</str<strong>on</strong>g> findings c<strong>on</strong>firm those <str<strong>on</strong>g>of</str<strong>on</strong>g> other recent meta-analyses <str<strong>on</strong>g>of</str<strong>on</strong>g> distance<br />
educati<strong>on</strong> programs, and provide a needed update to the meta-analysis focused <strong>on</strong> K-<strong>12</strong> students<br />
15 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
which was completed in 1998 just as the web-based systems were beginning to be studied in<br />
virtual schooling. <str<strong>on</strong>g>The</str<strong>on</strong>g> analysis showed that for the factors examined, distance learning did not<br />
outperform or underperform classroom instructi<strong>on</strong>. <str<strong>on</strong>g>The</str<strong>on</strong>g> number <str<strong>on</strong>g>of</str<strong>on</strong>g> studies was small, and many<br />
studies did not report detailed informati<strong>on</strong>, so the results should be viewed as indicati<strong>on</strong>s <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
tendencies rather than prescripti<strong>on</strong>s for practice.<br />
Implicati<strong>on</strong>s for Research and Practice<br />
<str<strong>on</strong>g>Distance</str<strong>on</strong>g> educati<strong>on</strong> as it has been implemented at the K-<strong>12</strong> level over the past decade has<br />
improved over time according to several measures: providing access to educati<strong>on</strong> and choice in<br />
course <str<strong>on</strong>g>of</str<strong>on</strong>g>ferings to increased numbers <str<strong>on</strong>g>of</str<strong>on</strong>g> students, <str<strong>on</strong>g>of</str<strong>on</strong>g>fering educati<strong>on</strong> to a larger range <str<strong>on</strong>g>of</str<strong>on</strong>g> grade<br />
levels and ability levels, using more interactive and widely accessible technologies, and leading<br />
students to academic success <strong>on</strong> a wider range <str<strong>on</strong>g>of</str<strong>on</strong>g> achievement instruments. <str<strong>on</strong>g>The</str<strong>on</strong>g> effect <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
distance educati<strong>on</strong> <strong>on</strong> learning may be moderated by several factors, existing as it does in a very<br />
complex web <str<strong>on</strong>g>of</str<strong>on</strong>g> educati<strong>on</strong>al, technological, and social dynamics. Factors such as the design <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
the distance learning system, the demands <str<strong>on</strong>g>of</str<strong>on</strong>g> the c<strong>on</strong>tent, the abilities and disabilities <str<strong>on</strong>g>of</str<strong>on</strong>g> the<br />
student, and the quality <str<strong>on</strong>g>of</str<strong>on</strong>g> the teacher are likely to be influential factors, as they are in<br />
c<strong>on</strong>venti<strong>on</strong>al educati<strong>on</strong>al enterprises. <str<strong>on</strong>g>The</str<strong>on</strong>g> c<strong>on</strong>sistency <str<strong>on</strong>g>of</str<strong>on</strong>g> the effects shown in the studies<br />
analyzed in this review suggest that as distance educati<strong>on</strong> is currently practiced, educators and<br />
other stakeholders can reas<strong>on</strong>ably expect learning in a well-designed distance educati<strong>on</strong><br />
envir<strong>on</strong>ment to be equivalent to learning in a well-designed classroom envir<strong>on</strong>ment.<br />
How will K-<strong>12</strong> distance educati<strong>on</strong> realize greater potential and maximize it effectiveness? How<br />
will designers and managers <str<strong>on</strong>g>of</str<strong>on</strong>g> K-<strong>12</strong> distance educati<strong>on</strong> programs make better decisi<strong>on</strong>s in order<br />
to design and deliver a more effective program? <str<strong>on</strong>g>The</str<strong>on</strong>g> answers lie in changes in the ways<br />
policymakers and researchers do their work in this complex c<strong>on</strong>text. In order for distance<br />
educati<strong>on</strong> to be evaluated, data must be collected and reported in detail. Such data collecti<strong>on</strong><br />
begins with identificati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> goals. Policymakers and evaluators must enter into a partnership in<br />
which comm<strong>on</strong> goals are identified, an evaluati<strong>on</strong> plan is acted <strong>on</strong>, and detailed reporting<br />
follows. Evaluati<strong>on</strong> must be seen as a tool to support policy setting and decisi<strong>on</strong> making (Means<br />
& Haertel 2004). It is no l<strong>on</strong>ger enough to ask whether distance educati<strong>on</strong> is effective, we need<br />
to understand why (Sabelli 2004). We need to know how to make it more effective, what factors<br />
c<strong>on</strong>tribute most to effectiveness, and in what c<strong>on</strong>texts the factors operate. Acquiring this<br />
knowledge requires c<strong>on</strong>sensus <strong>on</strong> a definiti<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> effectiveness that goes bey<strong>on</strong>d standardized<br />
tests, and a system for identifying and measuring factors that influence effectiveness. As Means<br />
and Haertel stress, “many studies <str<strong>on</strong>g>of</str<strong>on</strong>g> the effects <str<strong>on</strong>g>of</str<strong>on</strong>g> technology-supported innovati<strong>on</strong>s are hindered<br />
by a lack <str<strong>on</strong>g>of</str<strong>on</strong>g> measures <str<strong>on</strong>g>of</str<strong>on</strong>g> student learning commensurate with the initiative’s goals” (p. 99).<br />
One factor warranting special c<strong>on</strong>siderati<strong>on</strong> in assessing the effectiveness <str<strong>on</strong>g>of</str<strong>on</strong>g> virtual schooling is<br />
teacher quality. In classrooms, teacher effectiveness is a str<strong>on</strong>g determiner <str<strong>on</strong>g>of</str<strong>on</strong>g> differences in<br />
student learning, far outweighing differences in class size and heterogeneity (Darling-Hamm<strong>on</strong>d<br />
2000). Based <strong>on</strong> the similarities in student outcomes between distance and classroom learning,<br />
there is every reas<strong>on</strong> to expect that teacher preparati<strong>on</strong> is critical in distance educati<strong>on</strong>. However,<br />
there has been very little formal preparati<strong>on</strong> available addressing the unique nature <str<strong>on</strong>g>of</str<strong>on</strong>g> <strong>on</strong>line<br />
instructi<strong>on</strong> and very little time for teachers to develop their expertise as <strong>on</strong>line instructors. As<br />
16 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
pr<str<strong>on</strong>g>of</str<strong>on</strong>g>essi<strong>on</strong>al development becomes more comm<strong>on</strong> and expertise grows, student success is likely<br />
to grow as well.<br />
As sec<strong>on</strong>d factor that is growing in importance in K-<strong>12</strong> distance educati<strong>on</strong> is the emergence <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
virtual charter schools. By 2002, there were about 2000 charter schools nati<strong>on</strong>wide, and the No<br />
Child Left Behind Act allows public schools that “chr<strong>on</strong>ically fail” to make adequate yearly<br />
progress to be restructured as charter schools (Nels<strong>on</strong>, Rosenburg & Van Meter 2004, page 1).<br />
According to state department <str<strong>on</strong>g>of</str<strong>on</strong>g> educati<strong>on</strong> websites, there are now almost 100 virtual charter<br />
schools operating. This synthesis includes data from ten virtual charter schools, all <str<strong>on</strong>g>of</str<strong>on</strong>g> which<br />
performed at levels equivalent to n<strong>on</strong>-virtual public schools in their states. In c<strong>on</strong>trast, the 20004<br />
report <strong>on</strong> charter school achievement <strong>on</strong> the Nati<strong>on</strong>al Assessment <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g> Progress (Nels<strong>on</strong>,<br />
Rosenburg & Van Meter) provides evidence that charter schools overall are underperforming<br />
when compared to n<strong>on</strong>-charter public schools. Charter school students had significantly lower<br />
achievement in grades 4 and 8 math and reading, even when eligibility for free or reduced price<br />
lunch and urban locati<strong>on</strong> were factored into the comparis<strong>on</strong>. When minority status was used as a<br />
factor, it was found that black and Hispanic charter school students scored lower in 4 th grade<br />
math and reading, but the difference was not significant. <str<strong>on</strong>g>The</str<strong>on</strong>g> fact that virtual charter school<br />
students do not score lower that n<strong>on</strong>-virtual school students is a str<strong>on</strong>g indicator <str<strong>on</strong>g>of</str<strong>on</strong>g> the success <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
distance educati<strong>on</strong> for K-<strong>12</strong> learning.<br />
Practiti<strong>on</strong>ers and policymakers in K-<strong>12</strong> distance educati<strong>on</strong> are urged to use data-driven decisi<strong>on</strong><br />
making, and to do so they must be informed by experience and data must be available. In 2004,<br />
there have been fewer than ten years <str<strong>on</strong>g>of</str<strong>on</strong>g> accumulated experience and too little detailed research<br />
published <strong>on</strong> web-based distance educati<strong>on</strong> methods. <str<strong>on</strong>g>The</str<strong>on</strong>g> lack <str<strong>on</strong>g>of</str<strong>on</strong>g> detail in the research to date<br />
hinders thorough investigati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> the factors influencing practice, and limits what can be learned<br />
for the improvement <str<strong>on</strong>g>of</str<strong>on</strong>g> practice.<br />
C<strong>on</strong>clusi<strong>on</strong>s<br />
Learning using telecommunicati<strong>on</strong>s and general school learning can be very similar experiences.<br />
While distance learning as it is practiced in today’s virtual schools uses technology that is less<br />
than ten years old and advances rapidly, the literature has shown that a student’s educati<strong>on</strong> <strong>on</strong>line<br />
can be as effective as it is in a classroom, provided that a classroom with the appropriate course<br />
is accessible to the student. As the power <str<strong>on</strong>g>of</str<strong>on</strong>g> communicati<strong>on</strong> technology and educati<strong>on</strong>al<br />
technology grow, the skill <str<strong>on</strong>g>of</str<strong>on</strong>g> distance educators and designers will be challenged to provide<br />
experiences that use that power to provide an experience for students that improves <strong>on</strong> classroom<br />
instructi<strong>on</strong> with its limits <str<strong>on</strong>g>of</str<strong>on</strong>g> time and place. Research in K-<strong>12</strong> distance educati<strong>on</strong> is maturing<br />
al<strong>on</strong>gside the technology and those who use it, but current web-based distance educati<strong>on</strong> systems<br />
have <strong>on</strong>ly been studied for about the last five years at the K-<strong>12</strong> level, a very short time in which<br />
to build a body <str<strong>on</strong>g>of</str<strong>on</strong>g> literature.<br />
This meta-analysis represents a rigorous investigati<strong>on</strong> into the literature <strong>on</strong> K-<strong>12</strong> web-based<br />
distance educati<strong>on</strong> with attenti<strong>on</strong> <strong>on</strong> the factors likely to influence student performance. <str<strong>on</strong>g>The</str<strong>on</strong>g><br />
17 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
esult shows variati<strong>on</strong> in the degree <str<strong>on</strong>g>of</str<strong>on</strong>g> success students have experienced, and a need for more<br />
informati<strong>on</strong> if firm c<strong>on</strong>clusi<strong>on</strong>s are to be drawn. Blomeyer (2002) stated the recommendati<strong>on</strong><br />
well: “Support for additi<strong>on</strong>al pr<str<strong>on</strong>g>of</str<strong>on</strong>g>essi<strong>on</strong>ally designed and executed program evaluati<strong>on</strong>s and<br />
scientific educati<strong>on</strong>al research should be given a high priority in all public and private agencies<br />
supporting effective implementati<strong>on</strong> and use <str<strong>on</strong>g>of</str<strong>on</strong>g> <strong>on</strong>line learning in K-<strong>12</strong> learning communities”<br />
(page 10).<br />
As <str<strong>on</strong>g>of</str<strong>on</strong>g> spring, 2004, there were roughly 2,400 publicly-funded cyber-based charter schools and<br />
state and district virtual schools in 37 U.S. states, with an estimated 40,000 to 50,000 students<br />
participating in <strong>on</strong>line courses, according to Susan Patrick, Director <str<strong>on</strong>g>of</str<strong>on</strong>g> the U.S. Department <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
<str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>'s Office <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>al Technology (Fording 2004). With recent and c<strong>on</strong>tinued<br />
growth in virtual schools, virtual school leaders and policy makers will need a str<strong>on</strong>g research<br />
foundati<strong>on</strong> <strong>on</strong> which to base decisi<strong>on</strong>s.<br />
<str<strong>on</strong>g>The</str<strong>on</strong>g> Need for Prospective Study in Virtual Schooling<br />
An important step in the right directi<strong>on</strong> was taken in 2004 when the U.S. Department <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
<str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g> hosted an E-learning Summit to explore the status <str<strong>on</strong>g>of</str<strong>on</strong>g> K-<strong>12</strong> e-learning in the U.S. <str<strong>on</strong>g>The</str<strong>on</strong>g><br />
DOE Office <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>al Technology is showing leadership in including e-learning in the new<br />
Nati<strong>on</strong>al <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>al Technology Plan. Such initiatives begin to bring knowledge and expertise<br />
to more stakeholders, assist policymakers and practiti<strong>on</strong>ers in accessing informati<strong>on</strong>, and serve<br />
as a focal point for guiding future work that will improve outcomes across the spectrum.<br />
As a very recent innovati<strong>on</strong> in the sometimes slow-moving world <str<strong>on</strong>g>of</str<strong>on</strong>g> educati<strong>on</strong>, distance<br />
educati<strong>on</strong> has been shown over decades with every variety <str<strong>on</strong>g>of</str<strong>on</strong>g> technology to work effectively<br />
although it works in very different ways than classroom instructi<strong>on</strong> does, it meets different<br />
needs, and serves different audiences, having had far less time in which to mature. <str<strong>on</strong>g>The</str<strong>on</strong>g> literature<br />
c<strong>on</strong>tains reports <strong>on</strong> distance educati<strong>on</strong> programs in which student outcomes exceed those in<br />
c<strong>on</strong>venti<strong>on</strong>al classrooms, but in order to make use <str<strong>on</strong>g>of</str<strong>on</strong>g> such data in syntheses such as this <strong>on</strong>e,<br />
complete data need to be reported.<br />
Recommendati<strong>on</strong>s for K-<strong>12</strong> Online Learning Policy and Practice<br />
Policy-makers and practiti<strong>on</strong>ers should c<strong>on</strong>tinue to move forward in developing and<br />
implementing K-<strong>12</strong> distance educati<strong>on</strong> programs when those programs meet identified needs and<br />
when they are designed and managed as carefully as traditi<strong>on</strong>al educati<strong>on</strong> programs. <str<strong>on</strong>g>The</str<strong>on</strong>g> “no<br />
significant difference” result reported here and elsewhere lends c<strong>on</strong>fidence to distance educators<br />
that their <strong>on</strong>going efforts are likely to be effective. This synthesis, c<strong>on</strong>sidered together with<br />
current policy and recent research findings, dem<strong>on</strong>strates that students <str<strong>on</strong>g>of</str<strong>on</strong>g> many types and ages,<br />
can learn in many c<strong>on</strong>tent areas using the flexibility and choices afforded by distance educati<strong>on</strong>.<br />
Optimally, the research <strong>on</strong> K-<strong>12</strong> distance educati<strong>on</strong> would recommend specific practices that<br />
would lead to results that exceed those in c<strong>on</strong>venti<strong>on</strong>al educati<strong>on</strong> settings. <str<strong>on</strong>g>The</str<strong>on</strong>g> barriers that<br />
prevent such recommendati<strong>on</strong>s include:<br />
a limit <strong>on</strong> the educati<strong>on</strong>al expertise focused <strong>on</strong> distance educati<strong>on</strong> as an area <str<strong>on</strong>g>of</str<strong>on</strong>g> study<br />
18 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
a rather short-sighted view <str<strong>on</strong>g>of</str<strong>on</strong>g> the purposes <str<strong>on</strong>g>of</str<strong>on</strong>g> distance educati<strong>on</strong>, a lack <str<strong>on</strong>g>of</str<strong>on</strong>g> c<strong>on</strong>sensus<br />
about the goals <str<strong>on</strong>g>of</str<strong>on</strong>g> distance educati<strong>on</strong>, and an accompanying lack <str<strong>on</strong>g>of</str<strong>on</strong>g> evaluati<strong>on</strong> directed at<br />
assessing progress toward those goals<br />
a failure to take into account the complexity <str<strong>on</strong>g>of</str<strong>on</strong>g> systems in which distance educati<strong>on</strong><br />
operates<br />
a paucity <str<strong>on</strong>g>of</str<strong>on</strong>g> research and reporting that includes details sufficient for quantitative<br />
synthesis<br />
For distance educati<strong>on</strong> to add a prospective agenda to the generati<strong>on</strong>s <str<strong>on</strong>g>of</str<strong>on</strong>g> valuable retrospective<br />
study that currently guides the field, five major acti<strong>on</strong> recommendati<strong>on</strong> must be addressed by<br />
<strong>on</strong>line learning practiti<strong>on</strong>ers, <strong>on</strong>line learning district-level leadership, and Federal and State<br />
educati<strong>on</strong>al policy makers:<br />
1. First, the broader educati<strong>on</strong>al community needs to become better informed about K-<strong>12</strong><br />
<strong>on</strong>line learning and distance educati<strong>on</strong>, to foster better communicati<strong>on</strong> am<strong>on</strong>g the widest<br />
range <str<strong>on</strong>g>of</str<strong>on</strong>g> experts and practiti<strong>on</strong>ers who have the potential to c<strong>on</strong>tribute to advances in the<br />
field.<br />
This crucial informati<strong>on</strong>al campaign requires pr<str<strong>on</strong>g>of</str<strong>on</strong>g>essi<strong>on</strong>als working in distance educati<strong>on</strong><br />
in any capacity to network by participating in c<strong>on</strong>ferences, publishing articles and papers,<br />
and c<strong>on</strong>tributing to discussi<strong>on</strong>s locally and globally where people who are not involved in<br />
distance educati<strong>on</strong> can learn.<br />
2. Sec<strong>on</strong>d, the community <str<strong>on</strong>g>of</str<strong>on</strong>g> distance educati<strong>on</strong> policy makers, researchers, and<br />
practiti<strong>on</strong>ers should develop and articulate a l<strong>on</strong>g-range view <str<strong>on</strong>g>of</str<strong>on</strong>g> the intended and<br />
expected benefits <str<strong>on</strong>g>of</str<strong>on</strong>g> distance educati<strong>on</strong> and become advocates for suitably l<strong>on</strong>g-term<br />
studies <str<strong>on</strong>g>of</str<strong>on</strong>g> its effects.<br />
<str<strong>on</strong>g>The</str<strong>on</strong>g> list <str<strong>on</strong>g>of</str<strong>on</strong>g> potential benefits should be broad, and should be a close match to the benefits<br />
or “effects” anticipated for any educati<strong>on</strong>al experience. Curriculum c<strong>on</strong>tent should<br />
include a liberal educati<strong>on</strong> in which knowledge, skills, and dispositi<strong>on</strong>s are developed<br />
that successful students need in order to enjoy a full life in a democracy. But effects and<br />
benefits should also include academic literacies, technology skills, and academic<br />
standards.<br />
This list <str<strong>on</strong>g>of</str<strong>on</strong>g> crucial, performance-based knowledge, skills, and dispositi<strong>on</strong>s must serve as<br />
guide in the stages <str<strong>on</strong>g>of</str<strong>on</strong>g> design, implementati<strong>on</strong>, and evaluati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> programs. C<strong>on</strong>sensus is<br />
needed <strong>on</strong> the goals <str<strong>on</strong>g>of</str<strong>on</strong>g> distance educati<strong>on</strong>, and plans should follow to evaluate progress<br />
toward those goals. <str<strong>on</strong>g>Distance</str<strong>on</strong>g> educati<strong>on</strong> program directors should see researchers as<br />
partners in informing practice.<br />
3. Third, because educati<strong>on</strong> occurs in a dynamic c<strong>on</strong>text, and the rapid change in the<br />
technology used in distance educati<strong>on</strong> adds to the complexity, evaluati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> distance<br />
educati<strong>on</strong> programs needs to account for more <str<strong>on</strong>g>of</str<strong>on</strong>g> this complexity than has so far been the<br />
practice.<br />
19 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
A comm<strong>on</strong> “codebook” or heuristic descriptive system should be created and refined to<br />
ensure that outcomes from distance and <strong>on</strong>line learning programs can be accurately<br />
compared to other <strong>on</strong>line and distance programs and to face-to-face instructi<strong>on</strong>. A<br />
descriptive system supporting comparative analysis <str<strong>on</strong>g>of</str<strong>on</strong>g> all varieties <str<strong>on</strong>g>of</str<strong>on</strong>g> traditi<strong>on</strong>al and<br />
<strong>on</strong>line and distance learning delivery systems will dramatically increase both the<br />
generalizability <str<strong>on</strong>g>of</str<strong>on</strong>g> results and the synthesizability <str<strong>on</strong>g>of</str<strong>on</strong>g> research findings available to inform<br />
development, implementati<strong>on</strong> and instituti<strong>on</strong>alizati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> <strong>on</strong>line and distance learning<br />
programs.<br />
4. Finally, standards are needed for reporting the academic and programmatic outcomes <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
distance educati<strong>on</strong> programs. Many K-<strong>12</strong> distance educati<strong>on</strong> programs collect admirable<br />
amounts <str<strong>on</strong>g>of</str<strong>on</strong>g> data, and c<strong>on</strong>duct in-house analyses, but until there are standards set to guide<br />
the reporting <str<strong>on</strong>g>of</str<strong>on</strong>g> data, educati<strong>on</strong>al research will remain limited to examining results from<br />
<strong>on</strong>ly a small, enlightened subset <str<strong>on</strong>g>of</str<strong>on</strong>g> these programs.<br />
<str<strong>on</strong>g>The</str<strong>on</strong>g> acti<strong>on</strong>s recommended require coordinati<strong>on</strong> and leadership. Leadership should begin at the<br />
nati<strong>on</strong>al level and include pr<str<strong>on</strong>g>of</str<strong>on</strong>g>essi<strong>on</strong>al organizati<strong>on</strong>s like the North American Council <strong>on</strong> Online<br />
Learning and ISTE. <str<strong>on</strong>g>The</str<strong>on</strong>g> United States Department <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g> and the leading pr<str<strong>on</strong>g>of</str<strong>on</strong>g>essi<strong>on</strong>al<br />
organizati<strong>on</strong>s and groups should assume a leadership role organizing a nati<strong>on</strong>al distance learning<br />
and <strong>on</strong>line learning community <str<strong>on</strong>g>of</str<strong>on</strong>g> practice to work toward enacting these crucial acti<strong>on</strong><br />
recommendati<strong>on</strong>s.<br />
<str<strong>on</strong>g>Distance</str<strong>on</strong>g> educators bel<strong>on</strong>g to a wide variety <str<strong>on</strong>g>of</str<strong>on</strong>g> overlapping pr<str<strong>on</strong>g>of</str<strong>on</strong>g>essi<strong>on</strong>al groups and associati<strong>on</strong>s<br />
that have the potential to c<strong>on</strong>tribute to a powerful and effective coaliti<strong>on</strong>. <str<strong>on</strong>g>The</str<strong>on</strong>g> larger coaliti<strong>on</strong><br />
needed to weld a broader pr<str<strong>on</strong>g>of</str<strong>on</strong>g>essi<strong>on</strong>al c<strong>on</strong>sensus should serve as a central clearinghouse for<br />
informati<strong>on</strong> about K-<strong>12</strong> <strong>on</strong>line and distance educati<strong>on</strong>, a matchmaking service for programs and<br />
evaluators, and as an organizati<strong>on</strong>al focus for organizing nati<strong>on</strong>al efforts to support <strong>on</strong>line and<br />
distance learning policy and program development.<br />
Learning, progress, and data-driven decisi<strong>on</strong>s require the availability <str<strong>on</strong>g>of</str<strong>on</strong>g> relevant data. <str<strong>on</strong>g>The</str<strong>on</strong>g> K-<strong>12</strong><br />
distance educati<strong>on</strong> and <strong>on</strong>line learning communities certainly have the infrastructure for sharing<br />
that informati<strong>on</strong>. An adequate and uniform system for describing academic and programmatic<br />
outcomes within and across a variety <str<strong>on</strong>g>of</str<strong>on</strong>g> programs and instructi<strong>on</strong>al delivery systems, and<br />
uniform metrics and standards that can support comparis<strong>on</strong>s within and across the various<br />
delivery systems and instructi<strong>on</strong>al modalities.<br />
With ubiquitous availability <str<strong>on</strong>g>of</str<strong>on</strong>g> good informati<strong>on</strong> <strong>on</strong> the performance <str<strong>on</strong>g>of</str<strong>on</strong>g> all K-<strong>12</strong> educati<strong>on</strong>al<br />
programs and instructi<strong>on</strong>al systems, parents and practiti<strong>on</strong>ers, policymakers and nati<strong>on</strong>al<br />
political leadership can and will make the very best informed decisi<strong>on</strong>s about how to best<br />
educate and equip all our children for life and success during the ensuing twenty-first century.<br />
20 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
References<br />
References marked with an asterisk indicate studies used in the meta-analysis.<br />
Allen, M., Bourhis, J., Burrell, N., and Mabry, E. (2002). Comparing student satisfacti<strong>on</strong> with<br />
distance educati<strong>on</strong> to traditi<strong>on</strong>al classrooms in higher educati<strong>on</strong>: A meta-analysis. <str<strong>on</strong>g>The</str<strong>on</strong>g><br />
American Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Distance</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>, 16, 83 – 97.<br />
*Alaska Department <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g> and Early Development. (2003). Spring 2003 benchmarks,<br />
Delta/Greely School District - Delta Charter Cyber School. Juneau, AK: Author. Online at<br />
http://www.eed.state.ak.us/tls/assessment/AsmtVer/SchoolAsmtVerSuptSearch.cfm<br />
Arkansas Virtual School. (2003). 2002-2003 Testing Overview. Dardanelle, AR: author.<br />
Bangert-Drowns, R., Hurley, M., Wilkins<strong>on</strong>, B. (2004). <str<strong>on</strong>g>The</str<strong>on</strong>g> effects <str<strong>on</strong>g>of</str<strong>on</strong>g> school-based writing-tolearn<br />
interventi<strong>on</strong>s <strong>on</strong> academic achievement: a meta-analysis. Review <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>al<br />
Research, Spring 2003, 74 (1), 29-58.<br />
Barker, K., & Wendel, T. (2001). E-Learning: Studying Canada's Virtual Sec<strong>on</strong>dary Schools.<br />
Kelowna, BC: Society for the Advancement <str<strong>on</strong>g>of</str<strong>on</strong>g> Excellence in <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>. Online at<br />
http://www.excellenceineducati<strong>on</strong>.ca/pdfs/006.pdf.<br />
Bernard, R. M., Lou, Y., Abrami, P.C., Wozney, L., Borokhovski, E., Wallet, P.A., Wade, A.,<br />
and Fiset, M. (2003). How does distance educati<strong>on</strong> compare to classroom instructi<strong>on</strong>? A<br />
meta-analysis <str<strong>on</strong>g>of</str<strong>on</strong>g> the empirical literature. Presented as a Symposium at the Annual Meeting<br />
<str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>The</str<strong>on</strong>g> American <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>al Research Associati<strong>on</strong>, Chicago, IL.<br />
Bianchi, W. (2002). <str<strong>on</strong>g>The</str<strong>on</strong>g> Wisc<strong>on</strong>sin School <str<strong>on</strong>g>of</str<strong>on</strong>g> the Air: Success Story with Implicati<strong>on</strong>s.<br />
<str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>al Technology & Society 5 (1).<br />
Blomeyer, R. (2002). Virtual Schools and E-Learning in K-<strong>12</strong> Envir<strong>on</strong>ments: Emerging Policy<br />
and Practice, in NCREL Policy Issues, Issue 11, April 2002. Naperville, IL: North Central<br />
Regi<strong>on</strong>al <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>al Laboratory. Online at<br />
http://www.ncrel.org/policy/pubs/pdfs/pivol11.pdf.<br />
Bolt<strong>on</strong>, J. (2002). Web-based <str<strong>on</strong>g>Distance</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>: Pedagogy, Epistemology, and Instructi<strong>on</strong>al<br />
Design. University <str<strong>on</strong>g>of</str<strong>on</strong>g> Saskatchewan. Online at<br />
http://www.usask.ca/educati<strong>on</strong>/coursework/802papers/boult<strong>on</strong>/boult<strong>on</strong>.pdf<br />
B<strong>on</strong>d, A. (2002). Learning Music Online: An Accessible Program for Isolated <strong>Student</strong>s.<br />
Kensingt<strong>on</strong> Park, SA: Australian Nati<strong>on</strong>al Training Authority. Online at<br />
http://www.ncver.edu.au/research/proj/nr1013.pdf.<br />
Bransford, J., Brown, A., & Cocking, R. eds. (1999). How People Learn. Washingt<strong>on</strong>, DC:<br />
Nati<strong>on</strong>al Academies Press.<br />
21 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
Brooks, J. & Brooks, M. (1993). <str<strong>on</strong>g>The</str<strong>on</strong>g> case for c<strong>on</strong>structivist classrooms. Alexandria, VA:<br />
Associati<strong>on</strong> for Supervisi<strong>on</strong> and Curriculum Development.<br />
Bruckman, A. (1998). Computer Supported Cooperative Work. <str<strong>on</strong>g>The</str<strong>on</strong>g> Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> Collaborative<br />
Computing 7: 47–86, 1998.<br />
Calder<strong>on</strong>i, J. (1998). Telesecundaria: Using TV to Bring <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g> to Rural Mexico. <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g><br />
and Technology Technical Notes Series: World Bank Human Development Network.<br />
http://wbln0018.worldbank.org/HDNet/HDdocs.nsf/C11FBFF6C1B77F9985256686006D<br />
C949/1635F1703FE053B385256754006D8C3F/$FILE/telesecundaria.pdf<br />
Cavanaugh, C. S. (2001). <str<strong>on</strong>g>The</str<strong>on</strong>g> effectiveness <str<strong>on</strong>g>of</str<strong>on</strong>g> interactive distance educati<strong>on</strong> technologies in K-<br />
<strong>12</strong> learning: A meta-analysis. Internati<strong>on</strong>al Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>al Telecommunicati<strong>on</strong>s,<br />
7(1), 73 – 88.<br />
* Colorado Department <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>. (2003a). Brans<strong>on</strong> Alternative School, Grades 1-6 School<br />
Accountability Report. Denver, CO: Author. Online at<br />
http://reportcard.cde.state.co.us/reportcard/pdf/2003_1750_0948_E.pdf.<br />
* Colorado Department <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>. (2003b). Brans<strong>on</strong> Alternative School, Grades 7-8 School<br />
Accountability Report. Denver, CO: Author. Online at<br />
http://reportcard.cde.state.co.us/reportcard/pdf/2003_1750_0948_M.pdf.<br />
* Colorado Department <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>. (2003c). C<strong>on</strong>necti<strong>on</strong>s Academy, Grades 1-6 School<br />
Accountability Report. Denver, CO: Author. Online at<br />
http://reportcard.cde.state.co.us/reportcard/pdf/2003_0880_1887_E.pdf.<br />
* Colorado Department <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>. (2003c). C<strong>on</strong>necti<strong>on</strong>s Academy, Grades 7-8 School<br />
Accountability Report. Denver, CO: Author. Online at<br />
http://reportcard.cde.state.co.us/reportcard/pdf/2003_0880_1887_M.pdf.<br />
C<strong>on</strong>zemius, A; Sandrock, P. (2003). Developing World Language Programs in Elementary<br />
Grades. Madis<strong>on</strong>, WI: Wisc<strong>on</strong>sin State Department <str<strong>on</strong>g>of</str<strong>on</strong>g> Public Instructi<strong>on</strong>. ED 480156.<br />
Darling-Hamm<strong>on</strong>d, L. (2000). Teacher quality and student achievement: a review <str<strong>on</strong>g>of</str<strong>on</strong>g> state policy<br />
evidence. In <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g> Policy Analysis Archives. (8)1. January 1, 2000. Glass, G. Ed.<br />
Dees, S. (1994). An Investigati<strong>on</strong> Of <str<strong>on</strong>g>Distance</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g> Versus Traditi<strong>on</strong>al Course Delivery<br />
Using Comparis<strong>on</strong>s Of <strong>Student</strong> Achievement Scores In Advanced Placement Chemistry<br />
And Percepti<strong>on</strong>s Of Teachers And <strong>Student</strong>s About <str<strong>on</strong>g>The</str<strong>on</strong>g>ir Delivery System (Satellite<br />
Course). Northern Illinois University: Unpublished dissertati<strong>on</strong>.<br />
Falck, A-K; Husu, J; Kr<strong>on</strong>lund, T., Kynaslahti, H., Salminen, J., Sal<strong>on</strong>en, M. (1997). Testing<br />
virtual classroom in the school c<strong>on</strong>text. <str<strong>on</strong>g>Distance</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>. 18, 2. Pg. 213.<br />
22 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
Fording, L. (2004). <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>, 21st Century-Style. Newsweek, March 20, 2004. Online at<br />
http://www.msnbc.msn.com/id/4633<strong>12</strong>6/<br />
Gershaw, D. (1989). Line <strong>on</strong> Life: Locus <str<strong>on</strong>g>of</str<strong>on</strong>g> C<strong>on</strong>trol. Online at<br />
http://www3.azwestern.edu/psy/dgershaw/lol/C<strong>on</strong>trolLocus.html<br />
* Goc Karp, G. and Woods, M. (2003). Wellness NutriFit Online Learning in Physical <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g><br />
for High School <strong>Student</strong>s. Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> Interactive Online Learning, 2(2).<br />
http://www.ncolr.org/jiol/archives/2003/fall/03/index.html<br />
Hedges, L., Shymansky, J., Woodworth, G. (1989). Modern methods <str<strong>on</strong>g>of</str<strong>on</strong>g> meta-analysis.<br />
Washingt<strong>on</strong>, DC: Nati<strong>on</strong>al Science Teachers Associati<strong>on</strong>.<br />
Hinnant, E. (1994). <str<strong>on</strong>g>Distance</str<strong>on</strong>g> learning using digital fiber optics: a study <str<strong>on</strong>g>of</str<strong>on</strong>g> student achievement<br />
and student percepti<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> delivery system quality. Mississippi State University:<br />
unpublished dissertati<strong>on</strong>.<br />
* Indiana Department <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>. (2004). School Snapshot: Irvingt<strong>on</strong> Community School.<br />
Indianapolis, IN: Author. Online at<br />
http://mustang.doe.state.in.us/SEARCH/snapshot.cfm?schl=1537.<br />
Jordan, A. (2002). An Investigati<strong>on</strong> into the <str<strong>on</strong>g>Effects</str<strong>on</strong>g> <str<strong>on</strong>g>of</str<strong>on</strong>g> Online Teaching and Learning <strong>on</strong><br />
Achievement <strong>Outcomes</strong> at the Sec<strong>on</strong>dary Level. Fayetteville State University: Unpublished<br />
dissertati<strong>on</strong>.<br />
Kearsley, G. (2000). Online <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>. Belm<strong>on</strong>t, CA: Wadsworth.<br />
Keegan, D. (1996). Foundati<strong>on</strong>s <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Distance</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>. L<strong>on</strong>d<strong>on</strong>: Routledge.<br />
Kozma, R., Zucker, A., Espinoza, C., McGhee, R., Yarnall, L., Zalles, D., Lewis, A. (2000). <str<strong>on</strong>g>The</str<strong>on</strong>g><br />
Online Course Experience: Evaluati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> the Virtual High School’s Third Year <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
Implementati<strong>on</strong>, 1999-2000. Arlingt<strong>on</strong>, VA: SRI.<br />
http://www.govhs.org/Images/SRIEvals/$file/SRIAnnualReport2000.pdf<br />
Kromrey, J. & Hogarty, K. (2002). Estimates <str<strong>on</strong>g>of</str<strong>on</strong>g> Variance Comp<strong>on</strong>ents in Random <str<strong>on</strong>g>Effects</str<strong>on</strong>g> Meta-<br />
Analysis: Sensitivity to Violati<strong>on</strong>s <str<strong>on</strong>g>of</str<strong>on</strong>g> Normality and Variance Homogeneity. American<br />
Statistical Associati<strong>on</strong> Joint Statistical Meeting, 2002, pp.1963-1968.<br />
Lipsey, B. & Wils<strong>on</strong>, D. (2001). Practical meta-analysis. Thousand Oaks, CA: SAGE<br />
Publicati<strong>on</strong>s, Inc.<br />
Machtmes, K. and Asher, J. W. (2000). A meta-analysis <str<strong>on</strong>g>of</str<strong>on</strong>g> the effectiveness <str<strong>on</strong>g>of</str<strong>on</strong>g> telecourses in<br />
distance educati<strong>on</strong>. <str<strong>on</strong>g>The</str<strong>on</strong>g> American Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Distance</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>, 14(1), 27 – 46.<br />
McGreal, R. (1994). Comparis<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> the Attitudes <str<strong>on</strong>g>of</str<strong>on</strong>g> Learners Taking Audiographic<br />
Telec<strong>on</strong>ferencing Courses in Sec<strong>on</strong>dary Schools in Northern Ontario. Interpers<strong>on</strong>al<br />
23 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
Computing and Technology, 2(4), 11-23.<br />
http://www.helsinki.fi/science/optek/1994/n4/mcgreal.txt<br />
Means, B. & Haertel, G., eds. (2004). Using technology evaluati<strong>on</strong> to enhance student learning.<br />
New York: Teachers College Press.<br />
Mills, S. (2002). School Isn’t A Place Anymore: An Evaluati<strong>on</strong> Of Virtual Greenbush Online<br />
Courses For High School <strong>Student</strong>s. University <str<strong>on</strong>g>of</str<strong>on</strong>g> Kansas: Schiefelbusch Institute for Life<br />
Span Studies. Online at http://media.lsi.ku.edu/research/vgeval/vgevaluati<strong>on</strong>report.pdf.<br />
* Minnesota Department <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>. (2003). 2003 Report Card: Cyber Village Academy.<br />
Minneapolis, MN: Author. Online at<br />
http://educati<strong>on</strong>.state.mn.us/ReportCard/2003/RCF402507010.pdf.<br />
* Mock, R. (2000). Comparis<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> Online Coursework to Traditi<strong>on</strong>al Instructi<strong>on</strong>. Michigan State<br />
University: unpublished thesis. http://hobbes.lite.msu.edu/~robmock/masters/abstract.htm<br />
Nati<strong>on</strong>al Associati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> State Boards <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>. (2001). Any time, any place, any path, any<br />
pace: Taking the lead <strong>on</strong> e-learning policy. Alexandria, VA: author.<br />
Nels<strong>on</strong>, F., Rosenburg, B. & Van Meter, N. (2004). Charter school achievement <strong>on</strong> the 2003<br />
nati<strong>on</strong>al assessment <str<strong>on</strong>g>of</str<strong>on</strong>g> educati<strong>on</strong>al progress. Washingt<strong>on</strong>, DC: American Federati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
Teachers. Online at http://www.aft.org/pubsreports/downloads/teachers/NAEPCharterSchoolReport.pdf.<br />
Rotter, J. B. (1989). Internal versus external c<strong>on</strong>trol <str<strong>on</strong>g>of</str<strong>on</strong>g> reinforcement: A case history <str<strong>on</strong>g>of</str<strong>on</strong>g> a<br />
variable. American Psychologist, 45,489-493.<br />
Ryan, W. (1996). <str<strong>on</strong>g>The</str<strong>on</strong>g> Effectiveness <str<strong>on</strong>g>of</str<strong>on</strong>g> Traditi<strong>on</strong>al vs. Audiographics Delivery in Senior High<br />
Advanced Mathematics Course. Journal <str<strong>on</strong>g>of</str<strong>on</strong>g> <str<strong>on</strong>g>Distance</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>/Revue de l'enseignement à<br />
distance (11)2. http://www.cade-aced.ca/en_pub.php.<br />
* Schollie, B. (2001). <strong>Student</strong> Achievement and Performance Levels in Online <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g><br />
Research Study. Edm<strong>on</strong>t<strong>on</strong>, Alberta: Alberta Online C<strong>on</strong>sortium.<br />
http://www.alberta<strong>on</strong>line.ab.ca/pdfs/AOCresearch_full_report.pdf<br />
Shachar, M. & Neumann, Y. (2003). Differences Between Traditi<strong>on</strong>al and <str<strong>on</strong>g>Distance</str<strong>on</strong>g> <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g><br />
Academic Performances: A meta-analytic approach. Internati<strong>on</strong>al Review <str<strong>on</strong>g>of</str<strong>on</strong>g> Research in<br />
Open and <str<strong>on</strong>g>Distance</str<strong>on</strong>g> Learning. October.<br />
Slavin, R. (2003) <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>al Psychology, 7 th Ed. <str<strong>on</strong>g>The</str<strong>on</strong>g>ory and Practice. Bost<strong>on</strong>, MA: Pears<strong>on</strong><br />
<str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>, Inc.<br />
Southern Regi<strong>on</strong>al <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g> Board. (2003). Essential principles <str<strong>on</strong>g>of</str<strong>on</strong>g> high-quality <strong>on</strong>line teaching:<br />
guidelines for evaluating K-<strong>12</strong> <strong>on</strong>line teachers. Atlanta, GA: author. Online at<br />
http://www.sreb.org/programs/EdTech/pubs/PDF/Essential_Principles.pdf.<br />
24 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
* Stevens, K. (1999). Two Canadian Approaches to Teaching Biology, Chemistry, Mathematics<br />
and Physics to Senior High School <strong>Student</strong>s in Virtual Classes. Paper presented at the<br />
Australasian Science <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g> Research Associati<strong>on</strong>.<br />
http://www.tellearn.mun.ca/pubs/virtual.html<br />
* Texas <str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g> Agency. (2003). 2002-2003 campus performance: Southwest Virtual<br />
Preparatory School. Austin, TX: Author. Online at<br />
http://www.tea.state.tx.us/perfreport/aeis/2003/campus.srch.html.<br />
Ungerleider, C.S. & Burns, T.C. (2003). A systematic review <str<strong>on</strong>g>of</str<strong>on</strong>g> the effectiveness and efficiency<br />
<str<strong>on</strong>g>of</str<strong>on</strong>g> networked ICT in educati<strong>on</strong>: A state <str<strong>on</strong>g>of</str<strong>on</strong>g> the field report to the Council <str<strong>on</strong>g>of</str<strong>on</strong>g> Ministers <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
<str<strong>on</strong>g>Educati<strong>on</strong></str<strong>on</strong>g>, Canada and Industry Canada. Unpublished report.<br />
* Washingt<strong>on</strong> Office <str<strong>on</strong>g>of</str<strong>on</strong>g> the Superintendent <str<strong>on</strong>g>of</str<strong>on</strong>g> Public Instructi<strong>on</strong>. (2003). Internet academy 2002-<br />
2003 WASL results. Olympia, WA: Author. Online at<br />
http://reportcard.ospi.k<strong>12</strong>.wa.us/Reports/summary.aspx?schoolId=1164&reportLevel=Scho<br />
ol.<br />
Wedemeyer, C.A. (1981). Learning at the back door: Reflecti<strong>on</strong>s <strong>on</strong> n<strong>on</strong> traditi<strong>on</strong>al learning in<br />
the lifespan. Madis<strong>on</strong>: University <str<strong>on</strong>g>of</str<strong>on</strong>g> Wisc<strong>on</strong>sin Press.<br />
* Wisc<strong>on</strong>sin Department <str<strong>on</strong>g>of</str<strong>on</strong>g> Public Instructi<strong>on</strong>. (2003). Successful school guide: Wisc<strong>on</strong>sin<br />
Virtual Academy. Madis<strong>on</strong>, WI: Author. Online at<br />
http://data.dpi.state.wi.us/data/questi<strong>on</strong>s.asp?fullkey=011945040100&DN=Northern+Ozau<br />
kee&SN=Wisc<strong>on</strong>sin+Virtual+Academy&TYPECODE=6&CTY=45&ORGLEVEL=SC.<br />
Yasin, K. & Luberisse, Y. (1997). Meeting the Needs <str<strong>on</strong>g>of</str<strong>on</strong>g> a New Democracy: Multichannel<br />
Learning and Interactive Radio Instructi<strong>on</strong> in Haiti: A Case Study. Washingt<strong>on</strong>, DC:<br />
USAID. Online at http://ies.edc.org/pubs/book11.htm.<br />
25 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
Appendix A<br />
Coded Variables and Study Features in the Codebook<br />
A. Identificati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> studies<br />
1. Study number (“study”).<br />
2. Finding/hypothesis number (“finding”).<br />
3. Author name (“author”). Last name <str<strong>on</strong>g>of</str<strong>on</strong>g> first author.<br />
4. Year <str<strong>on</strong>g>of</str<strong>on</strong>g> publicati<strong>on</strong> (“year”).<br />
5. Number <str<strong>on</strong>g>of</str<strong>on</strong>g> findings/hypotheses within study (“number”).<br />
6. Country (“country”).<br />
Unspecified=0,<br />
USA=1,<br />
Canada=2,<br />
Mexico/Central America/South America=3,<br />
Europe=4,<br />
Asia=5,<br />
Africa=6,<br />
Australia/Pacific=7,<br />
Multinati<strong>on</strong>al=8,<br />
Other=9.<br />
7. Grade level <str<strong>on</strong>g>of</str<strong>on</strong>g> students (“grade”).<br />
Unspecified=00,<br />
grades 1-<strong>12</strong> use 01 to <strong>12</strong>,<br />
mixed primary (K-2) =13,<br />
mixed intermediate (3-5) =14,<br />
mixed middle (6-8) =15,<br />
mixed high (9-<strong>12</strong>) =16,<br />
K-<strong>12</strong>=17,<br />
other=18.<br />
8. School type (“school”).<br />
Unspecified=0,<br />
Public district sp<strong>on</strong>sored=1,<br />
Public state sp<strong>on</strong>sored=2,<br />
Private=3,<br />
Other=4,<br />
Charter=5,<br />
Combinati<strong>on</strong>=6.<br />
26 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
9. C<strong>on</strong>tent area (“c<strong>on</strong>tent”).<br />
Unspecified=0,<br />
Reading/Language arts=1,<br />
Mathematics=2,<br />
Social studies=3,<br />
Science=4,<br />
Computers/Technology=5,<br />
Foreign language=6,<br />
Arts=7,<br />
Physical educati<strong>on</strong>=8,<br />
Other=9,<br />
Writing=10.<br />
10. Type <str<strong>on</strong>g>of</str<strong>on</strong>g> publicati<strong>on</strong> (“publicati<strong>on</strong>”).<br />
Published journal article=1,<br />
Journal article in press=2,<br />
Book chapter=3,<br />
Report=4,<br />
Dissertati<strong>on</strong>=5,<br />
C<strong>on</strong>ference paper=6.<br />
B. <str<strong>on</strong>g>Distance</str<strong>on</strong>g> learning features<br />
1. Durati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> distance learning experience (“durati<strong>on</strong>”).<br />
Less than <strong>on</strong>e semester=1,<br />
One semester=2,<br />
More than <strong>on</strong>e semester=3.<br />
2. Frequency <str<strong>on</strong>g>of</str<strong>on</strong>g> distance learning experience (“frequency”).<br />
Unspecified=0,<br />
From 5 to 7 days per week=1,<br />
From 1 to 4 days per week=2,<br />
From 1 to 3 days per m<strong>on</strong>th=3,<br />
Less than m<strong>on</strong>thly=4.<br />
3. Instructi<strong>on</strong>al role <str<strong>on</strong>g>of</str<strong>on</strong>g> distance learning (“role”).<br />
Unspecified=0,<br />
Full-time educati<strong>on</strong>al program=1,<br />
Courses to supplement an educati<strong>on</strong>al program or partial educati<strong>on</strong>al program=2,<br />
Supplement to a specific course=3.<br />
4. Number <str<strong>on</strong>g>of</str<strong>on</strong>g> distance learning sessi<strong>on</strong>s (“dlnumber”).<br />
Unspecified=0,<br />
List number <str<strong>on</strong>g>of</str<strong>on</strong>g> sessi<strong>on</strong>s.<br />
5. Durati<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> distance learning sessi<strong>on</strong>s (“dldurati<strong>on</strong>”).<br />
Unspecified=0,<br />
27 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
List average minutes per sessi<strong>on</strong>.<br />
6. Pacing <str<strong>on</strong>g>of</str<strong>on</strong>g> distance learning instructi<strong>on</strong> (“pacing”).<br />
Unspecified=0,<br />
Completely self-paced=1,<br />
<strong>Student</strong> sets pace within instructor-determined parameters=2,<br />
Pacing completely specified by program or instructor=3.<br />
7. Instructor role (“instructrole”).<br />
Unspecified=0,<br />
Fully moderated=1,<br />
N<strong>on</strong>-moderated=2,<br />
Combinati<strong>on</strong>=3,<br />
Other=4.<br />
8. Timing <str<strong>on</strong>g>of</str<strong>on</strong>g> interacti<strong>on</strong>s (“timing”).<br />
Unspecified=0,<br />
Synchr<strong>on</strong>ous=1,<br />
Asynchr<strong>on</strong>ous=2,<br />
Combinati<strong>on</strong>=3,<br />
Other =4.<br />
8. Type <str<strong>on</strong>g>of</str<strong>on</strong>g> interacti<strong>on</strong>s (“interacti<strong>on</strong>”).<br />
Unspecified=0,<br />
<strong>Student</strong>—c<strong>on</strong>tent=1,<br />
<strong>Student</strong>—instructor=2,<br />
<strong>Student</strong>—student=3,<br />
<strong>Student</strong>—others=4,<br />
Combinati<strong>on</strong>=5,<br />
Other=6.<br />
C. Instructor/program features<br />
1. Amount <str<strong>on</strong>g>of</str<strong>on</strong>g> teacher preparati<strong>on</strong> in distance learning (“instructprep”).<br />
Unspecified=0,<br />
List hours <str<strong>on</strong>g>of</str<strong>on</strong>g> preparati<strong>on</strong>.<br />
2. Amount <str<strong>on</strong>g>of</str<strong>on</strong>g> teacher experience in distance learning (“instructexp”).<br />
Unspecified=0,<br />
List years <str<strong>on</strong>g>of</str<strong>on</strong>g> experience.<br />
3. Qualificati<strong>on</strong>s <str<strong>on</strong>g>of</str<strong>on</strong>g> teacher in the teaching field (“instructqual”).<br />
Unspecified=0,<br />
Certified in c<strong>on</strong>tent area=1,<br />
Certified but teaching out <str<strong>on</strong>g>of</str<strong>on</strong>g> field=2,<br />
Alternative or provisi<strong>on</strong>al certificati<strong>on</strong>=3,<br />
Uncertified=4,<br />
28 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
Other=5.<br />
4. Setting <str<strong>on</strong>g>of</str<strong>on</strong>g> students during distance learning (“setting”).<br />
Unspecified=0,<br />
Home=1,<br />
School=2,<br />
Other=3,<br />
Combinati<strong>on</strong>=4.<br />
D. Study quality features<br />
1. <strong>Student</strong> sample size (“sample”). Actual sample size.<br />
2. Measure <str<strong>on</strong>g>of</str<strong>on</strong>g> academic outcome (“achmeasure”).<br />
Standardized test=1,<br />
Researcher-made test=2,<br />
Teacher-made test=3,<br />
Other=4.<br />
3. Testing sequence (“testseq”).<br />
Unspecified=0,<br />
Pre-post=1,<br />
Post <strong>on</strong>ly=2,<br />
Other=3.<br />
4. Pretest equivalency (“preequiv”). Have the initial differences between groups been<br />
accounted for?<br />
Unspecified=0,<br />
Statistical c<strong>on</strong>trol (ANCOVA, regressi<strong>on</strong>)=1,<br />
Random assignment=2,<br />
Statistical c<strong>on</strong>trol and random assignment=3,<br />
Gain scores=4,<br />
Other=5.<br />
5. Reported reliability <str<strong>on</strong>g>of</str<strong>on</strong>g> measures (“reliability”).<br />
Unspecified=00,<br />
Actual reliability statistic.<br />
6. Effect size coefficient (“effsize”).<br />
Actual coefficient.<br />
7. Statistics used in determining effect size. (“esstats”).<br />
Means=1,<br />
t-value=2,<br />
F-value=3,<br />
Chi-square=4,<br />
Other=5.<br />
29 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
8. Weight (“weight”).<br />
One divided by the actual number <str<strong>on</strong>g>of</str<strong>on</strong>g> findings/hypotheses in the study.<br />
E. Sources <str<strong>on</strong>g>of</str<strong>on</strong>g> Invalidity<br />
1. Type <str<strong>on</strong>g>of</str<strong>on</strong>g> Design (“design”).<br />
Quasi-experimental/n<strong>on</strong>randomized <strong>on</strong>e group pretest-posttest=1,<br />
N<strong>on</strong>randomized static-group comparis<strong>on</strong>=2,<br />
N<strong>on</strong>randomized pre-post c<strong>on</strong>trol group=3,<br />
Time series=4,<br />
Randomized posttest-<strong>on</strong>ly c<strong>on</strong>trol group=5,<br />
Randomized pre-post c<strong>on</strong>trol group=6,<br />
Other=7.<br />
2. History (“history”). C<strong>on</strong>trol for specific events occurring between the first and sec<strong>on</strong>d<br />
measurement in additi<strong>on</strong> to the experimental variable.<br />
Adequately c<strong>on</strong>trolled by design=1,<br />
Definite weakness <str<strong>on</strong>g>of</str<strong>on</strong>g> design=2,<br />
Possible source <str<strong>on</strong>g>of</str<strong>on</strong>g> c<strong>on</strong>cern=3,<br />
Not a relevant factor=4.<br />
3. Maturati<strong>on</strong> (“maturati<strong>on</strong>”). C<strong>on</strong>trol for processes within the participants operating as a<br />
functi<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> the passage <str<strong>on</strong>g>of</str<strong>on</strong>g> time.<br />
Are there processes within participants operating as a functi<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> the passage <str<strong>on</strong>g>of</str<strong>on</strong>g> time,<br />
such as growing older or more tired, that might account for changes in the dependent<br />
measure?<br />
Adequately c<strong>on</strong>trolled by design=1,<br />
Definite weakness <str<strong>on</strong>g>of</str<strong>on</strong>g> design=2,<br />
Possible source <str<strong>on</strong>g>of</str<strong>on</strong>g> c<strong>on</strong>cern=3,<br />
Not a relevant factor=4.<br />
4. Testing (“testing”). C<strong>on</strong>trol for the effect <str<strong>on</strong>g>of</str<strong>on</strong>g> taking a test up<strong>on</strong> the scores <str<strong>on</strong>g>of</str<strong>on</strong>g> a sec<strong>on</strong>d<br />
testing.<br />
Adequately c<strong>on</strong>trolled by design=1,<br />
Definite weakness <str<strong>on</strong>g>of</str<strong>on</strong>g> design=2,<br />
Possible source <str<strong>on</strong>g>of</str<strong>on</strong>g> c<strong>on</strong>cern=3,<br />
Not a relevant factor=4.<br />
5. Instrumentati<strong>on</strong> (“instrument”). C<strong>on</strong>trol for changes in calibrati<strong>on</strong> or observers' scores<br />
that produce changes in the obtained measurement.<br />
Adequately c<strong>on</strong>trolled by design=1,<br />
Definite weakness <str<strong>on</strong>g>of</str<strong>on</strong>g> design=2,<br />
Possible source <str<strong>on</strong>g>of</str<strong>on</strong>g> c<strong>on</strong>cern=3,<br />
Not a relevant factor=4.<br />
30 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
6. Statistical Regressi<strong>on</strong> (“regressi<strong>on</strong>”). C<strong>on</strong>trol for group selecti<strong>on</strong> based <strong>on</strong> their extreme<br />
scores.<br />
Adequately c<strong>on</strong>trolled by design=1,<br />
Definite weakness <str<strong>on</strong>g>of</str<strong>on</strong>g> design=2,<br />
Possible source <str<strong>on</strong>g>of</str<strong>on</strong>g> c<strong>on</strong>cern=3,<br />
Not a relevant factor=4.<br />
7. Selecti<strong>on</strong> Bias (“selecti<strong>on</strong>”). C<strong>on</strong>trol for biases resulting in the differential selecti<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
comparis<strong>on</strong> groups.<br />
Adequately c<strong>on</strong>trolled by design=1,<br />
Definite weakness <str<strong>on</strong>g>of</str<strong>on</strong>g> design=2,<br />
Possible source <str<strong>on</strong>g>of</str<strong>on</strong>g> c<strong>on</strong>cern=3,<br />
Not a relevant factor=4.<br />
8. Mortality (“mortality”). C<strong>on</strong>trol for differential loss <str<strong>on</strong>g>of</str<strong>on</strong>g> participants from the<br />
experimental and c<strong>on</strong>trol groups.<br />
Adequately c<strong>on</strong>trolled by design=1,<br />
Definite weakness <str<strong>on</strong>g>of</str<strong>on</strong>g> design=2,<br />
Possible source <str<strong>on</strong>g>of</str<strong>on</strong>g> c<strong>on</strong>cern=3,<br />
Not a relevant factor=4.<br />
9. Selecti<strong>on</strong>-Maturati<strong>on</strong> Interacti<strong>on</strong> (“selectmatur”). C<strong>on</strong>trol for interacti<strong>on</strong> between<br />
extraneous factors such as history, maturati<strong>on</strong>, or testing and the specific selecti<strong>on</strong><br />
differences that distinguish the experimental and c<strong>on</strong>trol groups.<br />
Adequately c<strong>on</strong>trolled by design=1,<br />
Definite weakness <str<strong>on</strong>g>of</str<strong>on</strong>g> design=2,<br />
Possible source <str<strong>on</strong>g>of</str<strong>on</strong>g> c<strong>on</strong>cern=3,<br />
Not a relevant factor=4.<br />
10. Reactive or Interacti<strong>on</strong> Effect <str<strong>on</strong>g>of</str<strong>on</strong>g> Testing (“testeff”). C<strong>on</strong>trol for the influence <str<strong>on</strong>g>of</str<strong>on</strong>g> pretesting<br />
<strong>on</strong> the participants' resp<strong>on</strong>siveness to the experimental variable, making the<br />
results for a pre-tested populati<strong>on</strong> unrepresentative <str<strong>on</strong>g>of</str<strong>on</strong>g> the effects <str<strong>on</strong>g>of</str<strong>on</strong>g> the experimental<br />
variable for the unpre-tested universe from which the participants were selected.<br />
Adequately c<strong>on</strong>trolled by design=1,<br />
Definite weakness <str<strong>on</strong>g>of</str<strong>on</strong>g> design=2,<br />
Possible source <str<strong>on</strong>g>of</str<strong>on</strong>g> c<strong>on</strong>cern=3,<br />
Not a relevant factor=4.<br />
11. Interacti<strong>on</strong> <str<strong>on</strong>g>of</str<strong>on</strong>g> Selecti<strong>on</strong> Biases and Treatment (“biastreat”). C<strong>on</strong>trol for selective<br />
factors up<strong>on</strong> which sampling was based which interact differentially with the<br />
experimental variable.<br />
Adequately c<strong>on</strong>trolled by design=1,<br />
Definite weakness <str<strong>on</strong>g>of</str<strong>on</strong>g> design=2,<br />
Possible source <str<strong>on</strong>g>of</str<strong>on</strong>g> c<strong>on</strong>cern=3,<br />
Not a relevant factor=4.<br />
31 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32
<strong>12</strong>. Reactive <str<strong>on</strong>g>Effects</str<strong>on</strong>g> <str<strong>on</strong>g>of</str<strong>on</strong>g> Experimental Arrangements (“effexper”). C<strong>on</strong>trol for effects <str<strong>on</strong>g>of</str<strong>on</strong>g> the<br />
experimental setting that would preclude generalizing about the effect <str<strong>on</strong>g>of</str<strong>on</strong>g> the<br />
experimental variable up<strong>on</strong> pers<strong>on</strong>s being exposed to it in n<strong>on</strong>experimental settings.<br />
Adequately c<strong>on</strong>trolled by design=1,<br />
Definite weakness <str<strong>on</strong>g>of</str<strong>on</strong>g> design=2,<br />
Possible source <str<strong>on</strong>g>of</str<strong>on</strong>g> c<strong>on</strong>cern=3,<br />
Not a relevant factor=4.<br />
13. Multiple-Treatment Interference (“multtreat”) C<strong>on</strong>trol for n<strong>on</strong>erasable effects <str<strong>on</strong>g>of</str<strong>on</strong>g><br />
previous treatments applied to the same participants.<br />
Adequately c<strong>on</strong>trolled by design=1,<br />
Definite weakness <str<strong>on</strong>g>of</str<strong>on</strong>g> design=2,<br />
Possible source <str<strong>on</strong>g>of</str<strong>on</strong>g> c<strong>on</strong>cern=3,<br />
Not a relevant factor=4.<br />
14. Statistical Power (“statpower”). Large enough sample size to reject the null hypothesis<br />
at a given level <str<strong>on</strong>g>of</str<strong>on</strong>g> probability, or estimate coefficients within reas<strong>on</strong>ably small<br />
margins <str<strong>on</strong>g>of</str<strong>on</strong>g> error. A sample <str<strong>on</strong>g>of</str<strong>on</strong>g> over 60 for groups such as classes or schools; a sample<br />
<str<strong>on</strong>g>of</str<strong>on</strong>g> over 100 for individuals.<br />
Probable threat (100 for individuals)=2.<br />
32 <str<strong>on</strong>g>of</str<strong>on</strong>g> 32