How Does Distance Education Compare to Classroom Instruction ...

Meta-Analysis of the Comparative Literature in Distance Education 3technical media; d) the provision of two-way communication; and e) the quasi-permanentabsence of learning groups. In characterizing DE, Keegan also distinguishes between “distanceteaching” and “distance learning.” It is a fair distinction that applies to all organized educationalevents. Since learning does not always follow from teaching it is also a useful way of discussingthe various elements—teaching and learning—that constitute a total educational setting.The media used in DE have undergone remarkable changes over the years. Taylor (2001)characterizes five generations of DE, largely defined with regard to the media, and thereby therange of instructional options, available at the time of their prevalence. The progression thatTaylor describes moves along a rough continuum of increased flexibility, interactivity, materialsdelivery and access, beginning in the early years of DE when it was called correspondenceeducation (i.e., the media were print and the post office), through broadcast radio and televisionand on to current manifestations of interactive multi-media, the Internet, access to Web-basedresources, computer-mediated communication, and most recently campus portals providingaccess to the complete range of university services and facilities at a distance. Across the historyof DE research, most of these media have been implicated in DE studies where comparisons aremade to what is often referred to as “traditional classroom-based instruction” or “face-to-faceinstruction.”Media and DE Comparison StudiesClark (1983, 1994) rightly criticized early media comparison studies on a variety of grounds, themost important of which is that the medium under investigation, the instructional method that isinextricably tied to it and the content of instruction form a confound that renders their relativecontribution to achieving instructional goals impossible to untangle. Clark goes on to argue thatit is the instructional method that is the “active ingredient,” not the medium—the medium issimply a neutral carrier of content and of method. In essence, he argues that any medium,appropriately applied, can fulfill the conditions for quality instruction, and so cost and accessshould form the decision criteria for media selection. Several notable rebuttals of Clark’sposition followed (Ullmer, 1994; Kozma, 1994). Kozma argued that Clark’s original assessmentwas based on “old non-interactive technologies” that simply carried method and content, where adistinction between these elements could be clearly drawn. More recent media uses, he added,involve highly interactive sets of events that occur between learners and teachers, among learners(e.g., collaborative learning), often within a constructivist framework, and even between learnersand non-human agents or tools, so that a consideration of discrete variables no longer makessense. The distinction here seems to be “media to support teaching” and “media to supportlearning,” completely in line with Keegan’s reference to distance teaching and distance learning.Cobb (1997) added an interesting wrinkle to the debate. He argued that under certaincircumstances, the efficiency of a medium or symbol system can be judged by how much of alearner’s cognitive work it performs for them. By this logic, some media, then, do haveadvantages over other media, since it is “easier” to learn some things with certain media thanwith others. The way to advance media design, according to Cobb, “is to model learner andmedium as distributed information systems, with principled, empirically determined distributionsof information storage and processing over the course of learning” (p. 33). According to thisargument, the medium becomes the tool of the learner’s cognitive engagement and not simply anindependent and neutral device for delivering content to them. It is what the learner does with a

Meta-Analysis of the Comparative Literature in Distance Education 4medium that counts, not so much what the teacher does. It is within this distinction betweenteaching and learning that differences in media types used in DE might arise.One of the differences between DE and media comparison studies is that distance education isnot a medium of instruction, per se, although it depends entirely on the availability of media fordelivery and communication (Keegan, 1996). DE can be non-interactive or highly interactive andmay, in fact, encompass one or many media types (e.g., print + video + computer-basedsimulations + computer conferencing) in the service of a wide range of instructional objectives.In the same way, classroom instruction may include a wide mix of media forms. So in a wellconceivedand executed comparative study, where all of these aspects are present in bothconditions, differences may relate more to the proximity of learner and teacher (i.e., one ofKeegan’s defining characteristics of DE) and the often present asynchronicity of communication,as well as the attendant issues of instructional design, student motivation, feedback andencouragement, direct and timely communication and perceptions of isolation that distancestudents have reported feeling (Bernard & Amundsen, 1989). Shale (1990) comments that “Insum, distance education ought to be regarded as education at a distance. All of what constitutesthe process of education when teacher and student are able to meet face-to-face also constitutesthe process of education when teacher and student are physically separated.” (p. 334)This, in turn, suggests that “good” DE applications and “good” classroom instruction should be,in principle, relatively equal to one another, regardless of the media used, especially if the mediaare used simply for the delivery of content. However, when the medium is placed in the hands oflearners, to make learning more constructive or more efficient, as suggested by Kozma andCobb, the balance of effect may shift. In fact, in DE, media may transform the learningexperience in ways that are unanticipated and not regularly available in face-to-face instructionalsituations. For example, the use of computer-mediated communication means students must usewritten forms of expression to communicate with one another in articulating and developingideas, in arguing contrasting viewpoints, in refining opinions, in settling disputes, and so on(Abrami & Bures, 1996). This use of written language and peer interaction allows for thepossibility of increased reflection (Hawkes, 2001), the development of writing skills(Winkelmann, 1995), and higher quality performance through peer modeling and mentoring(Lou, Deidic & Rosenfield, 2003; Lou & MacGregor, under review). The critical thinkingliterature goes so far as to suggest that activity of this sort promotes the development of higherorderthinking skills (Garrison, Anderson & Archer, 2001; McKnight, 2001).Why Do Comparative DE Studies?Is it necessary or even desirable, then, to continue to conduct studies that directly compare DEwith classroom teaching? Clark (2000), by exclusion, claims that it is not, saying, “… allevaluations should explicitly investigate the relative benefits of two but compatible types ofdistance education technologies found in every distance education program” (p. 4). Smith andDillon (1999) argue that comparative studies are still useful, but only when they are done in lightof a full analysis of media attributes and their hypothesized effects on learning, and when thesesame attributes are present and clearly articulated in the compared conditions. In the eyes ofSmith and Dillon, it is only under these circumstances that comparative studies can push forwardour understanding of the features of DE and classroom instruction that make them similar anddifferent. Unfortunately, as Smith and Dillon point out, this level of analysis and clearaccounting of the similarities and differences between treatment and control is not often reported

Meta-Analysis of the Comparative Literature in Distance Education 6particularly true in small-sample studies where power to reject the null hypothesis (and thus therisk of making Type II errors) is high. Third, differential sample sizes of individual studies makeit impossible to aggregate the results of different studies solely on the basis of their test statistics.So Russell’s work represents neither a sufficient overall test of the hypothesis of no differencenor an estimate of the magnitude of effects attributable to DE (i.e., a non-significant hypothesistest can yield a reasonable effect size in small-sample studies).Another widely cited report (Phipps & Merisotis, 1999) prepared for the American Federation ofTeachers and the National Education Association, and entitled “What’s the difference: A reviewof contemporary research on the effectiveness of distance education in higher education” appearsto contain a similar level of bias to Russell’s work, but for a different reason. In the words of theauthors, “While this review of original research does not encompass every study published since1990, it does capture the most important and salient of these works.” (p. 154). In fact, just over40 empirical investigations are cited to illustrate specific points that are made by the authors. Theproblem is, how can one judge importance or salience without carefully crafted inclusion andexclusion criteria? The bias that is risked, then, is one of selecting research, even unconsciously,to make one’s point, rather than accurately characterizing the state of the research literaturearound a given question. While one of the findings of the report may generally be true—that theliterature lacks rigor of methodology and reporting—the findings of the “questionableeffectiveness of DE” based on a select number of studies is no more credible than Russell’sclaim of non-significance based on everything that has ever been published. Somewhere betweenthese extremes lies evidence that can be taken as more representative of the true state of affairs inthe population. In addition, there have been a number of more or less extensive qualitativereviews of research (e.g., Berge & Mrozowski, 2001; Saba, 2000; Jung & Rha, 2000; Schlosser& Anderson, 1994; Moore & Thompson, 1990).Four quantitative syntheses that are specifically related to DE have been published (Shachar,2002; Allen, Bourhis, Burrell & Mabry, 2002; Cavanaugh, 2001; Machtmes & Asher, 2000).Meta-analysis or quantitative synthesis was developed by Gene Glass and his associates (Glass,McGaw & Smith, 1981) to enable the combining of studies with different sample sizes. Effectsize, the metric of a meta-analysis, is an unbiased index of standardized differences between atreatment and control group (it can also be expressed as a correlation coefficient), and can beaveraged in a way that test statistics cannot. Refinements (Hedges’ g) by Hedges and Olkins(1985) further reduce the bias resulting from differential sample sizes among studies. Thus in ameta-analysis, the outcome is not a probabilistic declaration of effectiveness or ineffectiveness,as is the case in testing the null hypothesis. It is an approach to estimating how much onetreatment differs from another, over a large set of similar studies. An additional advantage ofmeta-analysis is that moderator variables (i.e., coded study features) can also be investigated toexplore more detailed relationships that may exist in the data. A careful analysis of theaccumulated evidence allows one to explore what might be responsible for the variability infindings across media, instructional design, course features, students, settings, etc., as well asresearch methodology, therefore shedding light on some of the issues of media, method andexperimental confounds that Clark and others have pointed to. At the same time, failure in one'sability to explore these issues exposes the limitations in the existing research, both in quantityand quality, indicating directions for further inquiry.

Meta-Analysis of the Comparative Literature in Distance Education 8the effects of DE on student achievement? 3) What are some optimal conditions for effectivedistance education? 4) What is the general quality of the DE empirical research literature?The specific media of interaction and communication (e.g., CMC, e-mail).• The explicit use of instructional design procedures (e.g., systematic vs. unsystematic).• The instructional method (e.g., lecture, discussion, project-based learning) that is employed.• Synchronous and asynchronous forms of communication and interaction.• Demographics (who takes DE courses) and subject matter areas.In addition to these issues, the following questions will be addressed in the Discussion section:• Does an analysis of the literature suggest directions for future research?• What would characterize an optimal DE comparison study?• What are the strengths and weaknesses of meta-analysis?• Where will this meta-analysis go from here?MethodThis meta-analysis is a quantitative synthesis of empirical studies since 1985 that compared theeffects of distance education and traditional classroom-based instruction on student achievement,attitude, retention rate and other outcomes. The year 1985 was chosen as a cut-off date sinceelectronically mediated, interactive DE became widely available around that time. Theprocedures employed to conduct this quantitative synthesis are described below under thefollowing subheadings: data sources and inclusion/exclusion criteria, outcomes and effect sizeextraction, and study features coding and data analysis.Data Sources and Inclusion/Exclusion CriteriaThe working definition of DE builds on Nipper's (1989) model of “third-generation distancelearning,” as well as Keegan's (1996) synthesis of recent definitions. Linked historically todevelopments in technology, first generation DE refers to the early days of print-basedcorrespondence study. Characterized by the establishment of the Open University in 1969,second generation DE refers to the period when print materials were integrated with broadcastTV and radio, audio and videocassettes, and increased student support. Third generation DE washeralded by the invention of Hypertext and the rise in the use of teleconferencing (i.e., audio andvideo). To this, Taylor (2001) adds the “fourth generation,” characterized by flexible learning(e.g., CMC, Internet accessible courses) and the “fifth generation” (e.g., online interactivemultimedia, Internet-based access to WWW resources). Generations three, four and fiverepresent moves away from directed and non-interactive courses to those characterized by a highdegree of learner control and two-way communication, as well as group-oriented processes andgreater flexibility in learning. With new communication technologies in hand and renewedinterest in the convergence of DE and traditional education, this is an appropriate time to review

Meta-Analysis of the Comparative Literature in Distance Education 9the research on third, fourth and fifth generation DE. Our definition of DE for the inclusion ofstudies thus reads:• The semi-permanent separation (place and/or time) of learner and instructor during plannedlearning events.• The presence of planning and preparation of the learning materials, student support services,and the final recognition of course completion by an educational organization.• The provision of two-way media to facilitate dialogue and interaction between the studentsand the instructor and among students.To be included in this meta-analysis, each study had to meet the following inclusion/exclusioncriteria:1. Each study had to involve an empirical comparison of DE as defined in this meta-analysis(including satellite/TV/radio broadcast + telephone/e-mail, e-mail-based correspondence,text-based correspondence + telephone, web/audio/video-based two-way telecommunication)with face-to-face classroom instruction (including lectures, seminars, tutorials and laboratorysessions).2. DE with some face-to-face meetings (less than 50%) was included. However, studies whereelectronic media were used to supplement regular face-to-face classes with the teacherphysically present were excluded.3. Each study had to report measured outcomes for both experimental and control groups.Studies with insufficient data for effect size calculations (e.g., with means but no standarddeviations or no inferential statistics or no sample size) were excluded.4. All studies had to be publicly available or archived.5. Outcomes included: achievement; retention rate; attitudes toward course, technology used,subject matter or instructor; and other.6. All levels of learners (from kindergarten to adults, whether formal schooling or professionaltraining) were included.7. The studies were published or presented no earlier than 1985.8. Studies comparing DE (experimental) with national standards or norms, rather than a controlcondition, were excluded.9. Outcome measures had to be the same or comparable. If the study explicitly stated thatdifferent exams were used for the experimental and control groups, the study was excluded.10. The outcome measures had to reflect individual courses rather than whole programs. Thus,programs composed of many different courses, where no opportunity existed to analyzeconditions and the corresponding outcomes for individual treatments, were excluded.11. Any study with a k of 2 or less was excluded.

Meta-Analysis of the Comparative Literature in Distance Education 1012. When data about a particular study were available from different sources (e.g., journalarticle and dissertation), although only the published source is referenced, additional datafrom the other source were used to make coding study features more detailed and accurate.The studies used in this meta-analysis were located through a comprehensive search of publiclyavailable literature from 1985 to the present. Electronic searches were performed on thefollowing databases: ERIC, PsycInfo, ProQuest Digital Dissertations, ABI/Inform Global onProQuest, Canadian Research Index, Applied Science and Technology Index, Biosis, andCommunication Abstracts. Web searches were performed using Google, AlltheWeb and Teomasearch engines. A manual search was performed in Educational Technology Abstracts and inseveral distance learning journals, including The American Journal of Distance Education,Distance Education, Journal of Distance Education, Open Learning, and Journal ofTelemedicine and Telecare. In addition, the reference lists of several earlier reviews includingMoore and Thompson (1990); Cavanaugh (2001); Russell (1999); Machtmes and Asher (2000);and Allen, Bourhis, Burrell, and Mabry (2002) were reviewed for possible inclusions. Althoughsearch strategies varied depending on the tool used, generally search terms included: “distanceeducation,” “distance learning,” “open learning” or “virtual university,” AND (“traditional,”“lecture,” “face-to-face” or “comparison”).As of February 26, 2003, 2,262 research studies concerning DE and traditional classroom-basedinstruction have been examined and 1,010 potential includes retrieved. Each of the retrievedstudies was then read by two researchers for possible inclusion using the inclusion/exclusioncriteria. The initial inter-rater agreement as to inclusion was 89%. Any study that was consideredfor exclusion by one researcher was cross-checked by the other researcher. One hundred andfifty-seven (157) studies met all inclusion criteria and were included in this meta-analysis.Outcomes and Effect Size ExtractionEffect sizes were extracted and are reported, separately, for each major outcome category,including achievement, retention rate, student attitude towards course, student attitude towardstechnology used, student attitude towards subject matter, student attitude towards instructor andother (i.e., unclassified).The basic index for the effect size calculation is the mean of the experimental group (DE) minusthe mean of the control group (traditional face-to-face instruction) divided by the pooledstandard deviation.ES = X E − X Cs pooledEffect sizes from data in such forms as t-tests, F-tests, p-levels and frequencies were computedvia conversion formulas provided by Glass, McGaw and Smith (1981) and Hedges, Shymanskyand Woodworth (1989). When multiple achievement data were reported (e.g., assignments,midterm and final exams, GPAs, grade distributions), the final exam scores were used forcalculating the effect size. When there was more than one control group and they did not differconsiderably, the weighted average of the two conditions was used; if only one of the controlgroups could be considered “purely” control (i.e., classical face-to-face instructional mode),while others involved some elements of DE treatment (e.g., originating studio site), the formerwas used as the control group. In studies where there were two DE conditions and one control

Meta-Analysis of the Comparative Literature in Distance Education 11condition, the weighted average of the two DE conditions was used. In studies where instructionwas simultaneously delivered to an originating site and remote sites (e.g. two-wayvideoconferencing), the originating site was considered to be the control condition and theremote sites the DE condition.For attitude surveys, we used the average of all items falling under one type of outcome (e.g.,attitude towards subject matter) so that only one effect size was generated from each study foreach outcome.For studies that reported only a significance level, effect sizes were estimated (e.g., t = 1.96 for α= .05). When the direction of the effect was not available, we used an estimated effect size ofzero. When the direction was reported, a “midpoint” approach was taken to estimate arepresentative t-value (i.e., midpoint between 0 and the critical t-value for the sample size to besignificant; Sedlmeier & Gigerenzer, 1989). The unit of analysis was the independent studyfinding; multiple outcomes were sometimes extracted from the same study both within andbetween outcome types. Within outcome types (e.g., achievement), multiple outcomes wereextracted for different courses; repeated measures were not coded separately.Outcomes and effect sizes from each study were extracted by two researchers independently andthen compared for reliability. The intercoder agreement rate was 91% for the number of effectsizes extracted within a study and 95% for effect size calculation. In total, 510 independenteffects sizes were extracted from 157 studies to date with 40,495 students (achievementoutcomes).Study Features CodingDevelopment of the codebook was initiated from a review of a sample of 10 retrieved studies, aswell as several review pieces (e.g., Phipps & Merisotis, 1999), conceptual papers (e.g., Smith &Dillon, 1999), and critiques (e.g., Saba, 2000). Studies were first nomologically coded to identifysalient study features present in the literature in order to avoid researcher bias (Abrami, et al.1988). Through nomological coding, a comprehensive codebook was developed (see AppendixA), which included the following study feature categories: research design; accessibility (e.g.,technical support); media used (e.g., attributes); human factors (e.g., learner/instructordifferences); course design (e.g., learning tasks); and pedagogy (e.g., collaboration). Ofparticular interest in the analysis were the related constructs of one-way and two-waycommunication, interactivity and active engagement. Synchronous and asynchronouscommunication and the important issue of time on task is also considered, together with featuresrelated to instructional control and group interaction. To allow for a direct comparison of DE andtraditional classroom-based instruction, many of the study features were written in the followingform:1. DE more than control group2. DE reported/control group not reported3. DE equal to control condition4. Control reported/DE not reported5. DE less than control group999. Missing (no information on DE or control reported)

Meta-Analysis of the Comparative Literature in Distance Education 12“More than” was defined as 66% or more, “equal to” as 34% to 65%, and “less than” as 33% orless.To date, two-thirds of the studies have been coded by two coders independently and compared.Their initial coding agreement was 88.71%. Disagreements between two coders were resolvedthrough discussion and further review of the disputed studies. One-third of the studies have beencoded by only one coder. Final coding and verification will be completed by Summer of 2003.Data AnalysisHedges and Olkin’s (1985) homogeneity procedures were employed in aggregating andanalyzing the effect sizes for each outcome. The unit of analysis for each outcome was theindependent study finding. Each effect size was first corrected for bias and weighted by theinverse of its sampling variance, resulting in Hedges’ g. Thus, more weight was given to findingsthat were based on larger sample sizes. For one study (Hittleman, 2001) that has extremely largesample sizes (e.g., 1,000,000+), the control sample sizes were reduced to 3000 with theexperimental groups reduced proportionally. The treatment k was then proportionally weighted.This procedure was used to avoid overweighting by one study. Outlier analyses were performedusing the homogeneity statistics reduction method of Hedges and Olkin (1985).The weighted effect sizes were then aggregated to form an overall weighted mean estimate of thetreatment effect. The significance of the mean effect size was judged by its 95% confidenceinterval. A significantly positive (+) mean effect size indicates that the results favor DEconditions; a significantly negative (–) mean effect size indicates that the results favor traditionalclassroom-based instruction.To determine whether the findings for each outcome shared a common effect size, the set ofeffect sizes was tested for homogeneity by the homogeneity statistic (Q T ). When all findingsshare the same population effect size, Q T has an approximate χ 2 distribution with k – 1 degrees offreedom, where k is the number of effect sizes. If the obtained Q T value is larger than the criticalvalue, the findings are determined to be significantly heterogeneous, meaning that there is morevariability in the effect sizes than chance fluctuation would allow. Study features analyses werethen performed to identify potential moderating factors.In the study feature analyses, each coded study feature with sufficient variability was testedthrough two homogeneity statistics, between-class homogeneity (Q B ) and within-classhomogeneity (Q W ). Q B tests for homogeneity of effect sizes across classes. It has an approximatechi-square distribution with p – 1 degrees of freedom, where p is the number of classes. If Q B isgreater than the critical value, it indicates a significant difference among the classes of effectsizes. Q W indicates whether the effect sizes within each class are homogeneous. It has anapproximate chi-square distribution with m – 1 degrees of freedom, where m is the number ofeffect sizes in each class. If Q W is greater than the critical value, it indicates that the effect sizeswithin the class are heterogeneous. Data-analyses were conducted using two meta-analysissoftware, DSTAT (Johnson, 1989) and Comprehensive Meta-Analysis (Biostat, 2000).

Meta-Analysis of the Comparative Literature in Distance Education 13ResultsIn total, 510 independent effect sizes were extracted from 157 studies that compared the effectsof interactive DE with traditional classroom instruction on student achievement, retention rates,attitudes, and other outcomes. The results of the analysis are presented by research questions.Research question 1: How does DE compare to traditional courses with regard to studentachievement, attitude and retention (i.e., opposite of dropout)?The overall weighted mean effect sizes, standard error, 95% confidence interval andhomogeneity statistics for each outcome are shown in Table 1.Table 1. Overall weighted mean effect sizes for achievement, attitudes and retention outcomes.OutcomeHedges’gEffect SizeStandardError95% ConfidenceIntervalLowerLimitUpperLimitHomogeneity of ESQ-value df pAchievement (k = 248)N = 40,495Retention (k = 73)N = 38,306Attitude Towards Course(k = 66) N = 9,097Attitude Towards Technology(k = 24) N =2,329Attitude Towards SubjectMatter (k = 11) N = 918Attitude Towards Instructor(k = 29) N = 3,328+.0551* .0120 +.0316 +.0786 928.4968 247 < .05–.1034* .0219 –.1464 –.0604 148.0092 72 < .05–.0087 .0222 –.0523 +.0349 362.9793 65 < .05+.1498* .0435 +.0645 +.2350 133.5462 23 < .05–.1876* .0673 –.3197 –.0555 18.0678 10 > .05–.1720* .0360 –.2426* –.1014 67.8027 28 < .05*p < .05Achievement data. The overall range of individual effect sizes is large; the smallest effect size is–2.1652, while the largest effect size is +2.6607 (see Figure 1) Three outliers were identifiedusing the homogeneity statistics reduction method of Hedges and Olkin (1985). After removal ofthe three outliers (k = 248), the weighted mean effect size was g = +0.0551. The 95% confidenceinterval of the standard error is +0.0316 to +0.0787, indicating a small but significantly positiveeffect for interactive DE on student achievement as compared to traditional classroominstruction. Even after removing the outliers, however, the findings were still significantlyheterogeneous, suggesting the further investigation of moderator variables (see Table 1).

Meta-Analysis of the Comparative Literature in Distance Education 143Variability of Effect Sizes on AchievemenFigure 1. Variability of effect sizes for “Achievement.”21Hedges’ gHedges' g01 9 17 25 33 41 49 57 65 73 81 89 97 105 113 121 129 137 145 153 161 169 177 185 193 201 209 217 225 233 241-1-2-3Distribution Distribution of Effect of Sizes Sorted by by their their Magnitud MagnitudeRetention data. The weighted mean effect size for retention was –0.2313 based on 73independent findings. The sample size (in the millions) of one study was reduced proportionallyto a maximum of 3,000 to avoid over-weighting. After adjustment, the mean weighted effect sizewas –0.1034 indicating that the retention rate in DE courses was significantly lower than in faceto-faceclassrooms. Conversely, DE had a higher dropout rate. The homogeneity statisticindicated that the effect sizes were not homogeneous across the findings (see Figure 2).0.8Variability of Effect Sizes on RetentioFigure 2. Variability in effect sizes for “Retention.”0.60.40.2Hedges’ gHedges'g0-0.21 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73-0.4-0.6-0.8-1Distribution of Effect Sizes Sorted by their MagnitudeDistribution of Effect Sizes Sorted by their Magnitude

Meta-Analysis of the Comparative Literature in Distance Education 15Attitude data. Student attitudes toward various aspects of the DE versus classroom instructionappeared mixed. While no significant difference was found in “Attitude towards course”between the two conditions, there was a small positive effect on the outcome “Attitude towardstechnology” and a small negative effect on “Attitude towards subject matter” and “Attitudetowards instructor.” However, with the exception of “attitude towards subject matter” outcome,the findings for each of the other outcomes were significantly heterogeneous (see Figures 3 to 6).1.5Variability of Effect Sizes on Student Attitude Towards CoursFigure 3. Variability in effect sizes for “Attitude Towards Course.”10.5Hedges’ g Hedges’ gHedges' gHedges' g01 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65-0.5-1-1.5-2Distribution of Effect Sizes Sorted by their MagnitudeDistribution of Effect Sizes Sorted by their MagnitudeVariability of Effect Sizes on Student Attitude Towards TechnoloFigure 4. Variability in effect sizes for “Attitude Towards Technology.”21.510.501 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24-0.5-1-1.5Distribution of Effect Sizes Sorted by their MagnitudeDistribution of Effect Sizes Sorted by their Magnitude

Meta-Analysis of the Comparative Literature in Distance Education 160.6Variability of Effect Sizes on Student Attitude Towards Subject MattFigure 5. Variability in effect sizes for “Attitude Towards Subject Matter.”0.40.2Hedges’ g Hedges’ gHedges'gHedges'g0-0.2-0.4-0.61.510.51 2 3 4 5 6 7 8 9 10 11Distribution Distribution of Effect of Sizes Sorted by by their their MagnitudeVariability of Effect Sizes on Student Attitude towards InstructFigure 6. Variability in effect sizes for “Attitude Towards Instructor.”01 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29-0.5-1Distribution of Effect Sizes Sorted by their MagnitudeDistribution of Effect Sizes Sorted by their MagnitudeThe data for “Other Outcomes” are not described in detail here.Research question 2: What study features moderate the effects of DE on student achievement?In all, forty-four (44) outcome, methodology and substantive study features were coded.However, because of the large amount of missing data, many of these could not be investigatedeffectively (see Appendix A for statistics on all coded features). Therefore, only the study

Meta-Analysis of the Comparative Literature in Distance Education 17features for which there is adequate information available are analyzed here (i.e., k ≥ 10 studies).An analysis of the methodological features of the studies is presented at the end of the Resultssection. The following is an analysis of coded study features that appear under the heading of“Course Design,” “Demographics” and “Pedagogy.”Course design and pedagogy. Table 2 details the most interpretable results under the category“Course Design” and “Pedagogy” in the codebook (low effect sizes, even though they might besignificant and small sample sizes are excluded; a complete listing is in Appendix A).The DE conditions outperformed the Control when “systematic ID” was equal, and likewisewhen DE conditions were provided “course information in advance.” The majority of two-wayinteractive media used in DE had significantly positive effects on student achievement whencompared to classroom instruction that did not use the media: #23 “use of two-way audioconferencing,” #24 “use of two-way video conferencing,” #25 “use of computer mediatedcommunication,” #26 “use of e-mail,” # 27 “use of broadcast TV or videotape or audiotape,” and#29 “use of telephone” and #32 “provision for synchronous technically mediated communicationwith teacher.” Only one pedagogical feature was interpretable: #49 “Teacher/student contactencouraged.”Table 2. Conditions where DE was significantly more or less effective than classroom instruction.Study FeaturesHedges’gEffect SizeStandardErrorConfidenceIntervalsLower95%Upper95%z-valueTest of H 0#17 Systematic instructional design• DE = to Control (k = 24) +.2155 .0331 +.1507 +.2803 +6.5174 < .05#18 DE provided course information• In advance of course (k = 11) +.6350 .0649 +.5077 +.7624 +9.7835 < .05# 23 Two-way audio conferencing 1• DE used; Control did not (k = 33) +.1668 .0236 +.1205 +.2131 +7.061 < .05#24 Two-way video conferencing 2• DE used; Control did not (k = 12) +.1433 .0333 +.0779 +.2086 +4.298 < .05#25 Computer MediatedCommunication• DE used; Control did not (k = 22) + .2052 .0484 +.1103 +.3002 +4.239 < .05#26 e-mail• DE used more than Control (k = 9) –.2937 .0814 –.4535 –.1339 –3.608 < .05p#27 Use of one-way video(broadcast or recorded)• DE used; Control did not (k = 97)+.0632 .0205 +.0230 +.1033 +3.085 < .05

Meta-Analysis of the Comparative Literature in Distance Education 18#29 Use of telephone• DE = Control (k = 17) +.2618 .0575 +.1490 +.3746 +4.554 < .05#32 Synchronous communication• With the teacher (k = 137) +.0968 .0157 +.0661 +.1275 +6.178 < .05# 49 Teacher/student contactencouraged• DE more than Control (k = 11) +.3079 .0800 +.1508 +.4651 3.8476 < .051 Two-way audio, with or without video2 Two-way video, with or without audioDemographics. Effect sizes appeared varied for different types of students (see Table 3). DEconditions outperformed the control for K-12 grade students, graduate students, military trainees,and industry/business trainees. DE and control conditions performed equally for undergraduatestudents and professionals. Findings within each category were significantly heterogeneous.Table 3. Weighted mean effect sizes for different types of learners.Study FeaturesHedges’gEffect SizeStandardErrorConfidenceIntervalsLower95%Upper95%z-valueTest of H 0p#41 Type of DE Learners K-12 grade school (k = 22) +.1210 .0353 +.0517 +.1903 +3.4230 < .05 Undergraduate (k = 153) +.0311 .0181 –.0043 +.0665 +1.7239 > .05 Graduate (k = 29) +.0886 .0282 +.0333 +.1440 +3.1380 < .05 Military (k = 10) +.2050 .0558 +.0956 +.3145 +3.6752 < .05 Industry/business (k = 14) +.2027 .0815 +.0426 +.3627 +2.4858 < .05 Professionals (e.g., doctors)(k = 10)–.0525 .0428 –.1363 +.0314 –1.2264 > .05 Unspecified or more than onecategory (k = 10) +.0138 .0354 –.0556 +.0833 +.3908 > .05Table 4 presents the results in different subject areas. DE conditions outperformed the controlwhen subject matter involved “language arts or second languages” and “sciences orengineering.” DE and control conditions performed equally when subject matter involved“mathematics,” “humanities or social sciences” and “military training.” DE conditionsunderperformed the control when subject matter involved “business management.” Findingswithin each category were significantly heterogeneous.

Meta-Analysis of the Comparative Literature in Distance Education 19Table 4. Weighted mean effect sizes for different subject areas.Study FeaturesHedges’gEffect SizeStandardErrorConfidenceIntervalsLower95%Upper95%z-valueTest of H 0p#44 Subject Matter Mathematics (including statistics)(k = 25) –.0479 .0417 –.1297 +.0339 –1.1472 > .05• Languages (including languagearts and second languages)(k = 18)+.1539 .0459 +.0639 +.2439 +3.3511 < .05• Science and engineering (k =60 ) +.1331 .0282 +.0778 +.1884 +4.7156 < .05• Humanities and social sciences(k = 44 ) –.0046 .0313 –.0660 +.0567 –.1481 > .05 Business management (k = 41) –.1113 .0393 –.1883 –.0343 –2.8328 < .05• Military training (k = 11) +.0529 .0385 –.0225 +.1283 +1.3751 > .05 Unspecified, other or combined(k = 49)+.0921 .0209 +.0511 +.1331 +4.4049 < .05Table 5 presents the results of other demographic features. DE conditions outperformed thecontrol when DE instructors had experience with technologies used, “DE students’ ability” wereequal to control students’ ability, DE and control d “urban settings,” “attrition rate” between DEand control conditions were equal, DE students were older than or equal to the control and“gender ratio” between DE and control conditions was equal. When DE and control conditionsboth involved “rural settings,” control students outperformed DE students.Table 5. Other demographic conditions where DE was significantly more or less effective than classroominstructionConfidenceEffect SizeTest of HIntervals0Hedges’ Standard Lower UpperStudy Featuresz-value pg Error 95% 95%#37 Instructor experience withtechnology used• Yes (k =23)+.1055 .0392 +.0286 +.1823 +2.6918

Meta-Analysis of the Comparative Literature in Distance Education 20• DE older than control (k = 23)• DE = Control (k = 49)+.1921+.1075.0544.0261+.0854+.0563+.2988+.1587+3.5326+4.1132< .05< .05#48 Gender• DE = Control (k = 43) +.1433 .0333 +.0779 +.2086 +4.298 < .05Research question 3: What are some optimal conditions for effective distance education?Since the majority of significant and interpretable effects fell in the category of “media used,” anattempt was made to determine which combinations of media most influence achievement. To dothis, a composite index was created by combining study features 23 through 29 (#23 “two-wayaudio conferencing,” # 24 “two-way video conferencing,” #25 “use of CMC,” # 26 “use of e-mail,” #27 “one-way broadcast TV or videotape,” #28 “use of WEB-based materials,” #29 “useof the telephone” and # 30 “computer-based tutorials/simulations”) on the basis of their inclusionin DE studies (coded item “DE used, Control did not”). There were three groupings of mediause, one medium used, g = +0.0358, p > .05, k = 116; two media used, g = +0.1597, p < .05, k =66; and three media used, g = –0.0052, p > .05, k = 27. Studies that used two media, then, wereinvestigated. The following three combinations of media were identified: 1) “two-way audio” +“one-way broadcast TV or audiotape” (g = +0.3476, p < .05, k = 10); 2) “two-way audio” +“two-way video”—together representing standard videoconferencing, (g = +0.1524, p < .05, k =6); and 3) “one-way broadcast TV” + “use of telephone” (g = +0.1567, p < .05, k = 4).Research question 4: What is the general quality of the DE empirical research literature?Experimental design and methodology features that significantly moderated effect sizes include:“outcome measure source,” “type of publication,” “treatment duration,” “treatment proximity,”“instructor equivalence,” “student equivalence” and “material equivalence.” Effects were morepositive when achievement results were measured by “researcher-made tests” (g = +0.2252, k =24), “reported in published journal articles” (g = +0.0976, k = 91), based on “treatment durationof more than one semester” (g = +0.1322, k = 23), implemented “at a different time period” (g =+0.2367, k = 20), “taught by different instructors” (g = +0.2369, k = 22), student equivalence was“experimentally or statistically controlled” (g = +0.1896, k = 11; g = +0.1551, k = 30,respectively) and “different curriculum materials were utilized” (g = +0.1721, k = 31).In an attempt to characterize the overall quality of the comparative research literature of distanceeducation, an “index of quality” was created. The coded categories for six items (# 9 “type ofpublication,” #11 “treatment proximity,” # 13 “instructor equivalence,” #14 “studentequivalence,” #15 “equivalent time-on-task” and #16 “material equivalence”) were weighted ona scale from –1, 0 and +1 (i.e., where +1 was a positive feature and –1 was a negative feature).Missing values were weighted as 0. Sums were produced that ranged from –6 to +6 and thesewere grouped into three categories in the following way: +6 to +2 = high quality; +1 to –1 =medium quality; and –2 to –6 = low quality. When meta-analytic statistics were calculated onthese categories, the following resulted: high quality (g = +0.0071, k = 151, p > .05); mediumquality (g = +0.0565, k = 91, p < .05); and low quality (g = +0.4439, k = 5, p < .05). Thus, thelargest effect sizes are associated with the least methodological quality.Another way of judging the quality of the literature is in terms of what was codable and whatwas not. For the six methodological features (i.e., achievement data only), 36.96% of the studies

Meta-Analysis of the Comparative Literature in Distance Education 21were uncodable. For 30 of the other study features, 72.46% of the studies were uncodable. Thisindicates a literature that is incomplete, at best.DiscussionThe Quality of the DE LiteratureThe main purpose of this meta-analysis is to characterize the literature of DE in comparison toclassroom instruction, over various outcome measures, and to attempt to pinpoint the features ofDE that make it either more effective or less effective. An additional purpose is to assess thequality and the depth of the research literature upon which its findings are based. This discussionbegins with that assessment, because both the quality of studies and the depth of reportingimpinge upon all other aspects of the analysis.In the last few years, the research literature of DE has undergone intense scrutiny and has beenfound to be severely wanting in a variety of ways (Anglin & Morrison, 2000; Diaz, 2000;Perraton, 2000; Saba, 2000; Phipps & Merisotis, 1999). Among the issues that have been raisedare: a) lack of experimental control; b) lack of procedures for randomly selecting researchparticipants; c) lack of random assignment of participants to treatment conditions; d) poorlydesigned dependent measures that lack reliability and validity; and e) failure to account for avariety of variables related to the attitudes of students and instructors. While these flaws cannotbe addressed entirely through meta-analysis, their effects on the outcomes of research can beinvestigated, depending on the completeness of the reports that exist in the literature.There are two aspects to this problem. One is the methodological quality of studies and the otheris the completeness of the reporting. One whole section of the codebook deals withmethodological aspects of the studies that were reviewed. Our intent was not to exclude studiesthat had methodological weaknesses, like lack of random assignment or non-equivalentmaterials, but to code these features and examine how they affect the conclusions that can bedrawn from the studies. The results of the individual effect size calculations for methodologicalfeatures was presented in the Results section and are re-capped here.• Researcher-made test produced a positive effect (g = +0.2252, k = 23).• Studies in published journals produced a positive effect (g = +0.0976, k = 23).• Treatment duration > one semester produced a positive effect (g = +0.1322, k = 23).• DE/Control implemented at different times (g = +0.2367, k = 20).• DE/Control taught by different instructors produced a positive effect (g = +0.2369, k = 21).• Experimental/statistical control was positive (g = +0.1896, k = 29).• Different DE/Control materials used produced a positive effect (g = +0.1721, k = 30).As can be seen, some of these are positive and some are negative, so in order to characterize theliterature as a whole, an index of methodological quality was created from the nine coded studyfeatures. A description of how this was done appears in the Results section. Table 6 shows thefrequency and percentage of studies (achievement outcome) that were classified on the StudyQuality Index.

Meta-Analysis of the Comparative Literature in Distance Education 22Table 6. Frequency of Studies Classified on the Study Quality IndexCategory Scale Frequency Percentage Hedges’gHigh +2 to +6 152.0 61% +0.0071Medium +1 to –1 91.0 37% +0.0565Low –2 to –6 5.0 2% +0.4439Our overall analysis of methodological quality indicates that that the bulk of studies (rangingfrom –3 to +6 on this scale) are high or medium quality and that medium quality (k = 91)accounts for about the same average effect size as that of the overall literature (g = +0.0551).High quality studies (k = 152), by contrast, produce an effect size of +0.0071, suggesting thatbetter quality studies account for a smaller amount of variability in effect sizes than less welldesigned studies. Another way of saying this is that quality is inversely related to effect size.This fits with Clark’s assessment of the media literature (not specifically DE) and his conclusionthat confounding makes media comparison studies difficult to interpret. The effect size for lowquality studies is even more dramatic, but is based on only 5 outcomes. In all fairness, however,all of the numbers are positive, suggesting that methodological weaknesses do not bifurcatepositive and negative effect size. Another note, relative to the next paragraph, is that in thisanalysis 999s (missing values) were set to 0 so they would not contribute to the sum eitherpositively or negatively. If all of the 0s on the scale had been removed (studies with noinformation about methodology), a very different picture might have emerged. Furtherconsideration of the methodology question will follow as these data are explored further.The second issue relates to the completeness of the literature. Essentially, “Do researchers giveus enough information to judge the factors that contribute to or detract from effective practice?”The answer to this is an unequivocal NO: 72% of codable data relating to study features ismissing from the public record. It is often said, usually with regret, that the meta-analyticresearcher is a “prisoner of the previous efforts of primary researchers.” What is not reportedcannot be interpreted, retrospectively. For research to affect practice it must be trustworthy anddetailed, so that substantive conclusions go beyond simply whether the treatment works or not.This is certainly one of the most persistent problems in the comparative literature of DE and mayaccount for the minor extent to which it has affected pedagogical practices.While many of methodological issues are difficult to address in a field of practice, where theexternal validity of studies may either intentionally or inadvertently take precedence overinternal validity, there is no excuse for incomplete reporting. In particular, it is the descriptionsof “traditional classroom instruction” that are wanting. If one cannot discern what a DEcondition is being compared to, it is very difficult to come to any conclusion as to what an effectsize characterizing their difference means.Overall Findings from the DE Literature Related to Student AchievementIt is clear from the results of this meta-analysis that, on average, interactive distance education isas effective as traditional classroom instruction (g = +0.0551) in meeting objectives related tocontent acquisition (i.e., achievement). However, the variability across outcomes associated withaverage achievement is enormous, suggesting that the mean effect size is not truly arepresentative descriptor of the distribution. It is interesting to note that the distribution of 248effect sizes is nearly symmetrical (Figure 1), in terms of both magnitude and frequency. Wide

Meta-Analysis of the Comparative Literature in Distance Education 23symmetrical variability around a mean of nearly 0 strongly suggests that the answer to thequestion: “Is DE as effective as classroom instruction?” is that “it depends.” This is not aparticularly satisfying answer for the myriad of researchers, instructors and administratorsseeking to address theoretical and practical concerns about the nature of DE, but it is the bestoverall answer that the literature of DE has to offer at the present time. However, this perspectivegoes far beyond Russell’s claim (1999) of no significant difference because it supports the viewthat “not all DE is created equal”; it is likely that there are instructional practices and DEenvironments that are more conducive to learning, and others that are less so.The next question is of course, if it depends, “What does it depend on?”—what separateseffective DE practice from less effective DE practice? The general answer that emerges from thismeta-analysis is that effectiveness, at least in terms of student achievement, appears to depend onthe provision of communication and interactivity—the very features that earlier generations ofDE lacked, in comparison to classroom-based teaching. Four study features, all representingforms of interactivity (“two-way audio conferencing,” “two-way video conferencing,” “computermediated communication” and “the telephone”) appear to affect the overall quality of distanceeducation. In addition, it appears that one variable, the use of synchronous communicationbetween teacher and student, summarizes this general portrait as one of “transparency”—DE as itrecreates the conditions of the classroom. This finding is at odds with the results of Cavanaugh’s(2001) meta-analysis of DE studies in K–12 learning environments. With an overall weightedeffect size of +0.147 (Note: the current study produced similar results for K-12, g = +0.1210, k =22), she found substantially larger effects for telecommunications, defined as e-mail and theWeb, (ES =+0.488, k = 6) than for the use of two-way audio/video conferencing (ES = –0.011, k= 13).This current finding does raise an interesting question. If synchronous two-way learning andcommunication technologies best facilitate achievement in distance education applications,thereby making it more like face-to-face learning environments, why did this form of DEoutperform face-to-face instruction? After all, the above conclusions were drawn fromcomparisons of DE with traditional instruction, not with asynchronous forms of DE. There arethree classes of explanations that might shed further light on these findings. First, there may bemethodological flaws and confounds (e.g., selection bias, non-equivalent methods or materials,different instructors) that might account for these findings. Second, there may be novelty effects(i.e., Hawthorne Effect) of distance education courses and the technology aspect that is involvedin it. Third, there may be additional features of the DE treatments that have an effect. Thesemight include differences among learners (e.g., #41 K-12), more written communication anddialogue among students (e.g., #25 CMC), greater attention or more feedback from the instructor(e.g., #49, encouragement), etc. In future work it may be possible to untangle this finding andshed more light on just what matters in conducting effective DE.The public record has so little to say about asynchronous varieties of DE largely because of therelatively small number of studies that have been done with it. Only 15% of asynchronous studyfeatures could be coded, compared with 28% for the entire meta-analysis. Therefore, we cannotjudge the effectiveness of computer applications of text-based asynchronous DE, which comprisethe bulk of what is called “e-learning” and “Web-based instruction.” It may take a few yearsbefore an assessment of these recent entrants to DE can be made. In addition, greater bandwidth,

Meta-Analysis of the Comparative Literature in Distance Education 24allowing for streamed video and other media-oriented features, may make computer-based formsof DE as good as any other form.In terms of type of learners and the subject matter of DE and traditional instruction, there were anumber of interesting findings. Four groups of learners appeared to profit significantly from DE:K-12 students, graduate students, military personnel and employees in business and industry.The latter two produced effect sizes of greater than +0.20, but in both cases the number ofoutcomes was relatively small (k = 10 and 14, respectively). Two subject matter areas appear tostand out as benefiting from DE. Language instruction, including language arts and secondlanguage instruction, and science and engineering produced significant effect sizes that weregreater than +0.10. In the case of science and engineering, this finding is based on 60 outcomes.Future investigations within this literature may reveal interactions among these two studyfeatures and other features that are of interest.Neutral and Negative FactorsWhat factors either made little difference or detracted from the effectiveness of DE? Twovariables did not produce results where they might otherwise have been expected to. Teaching ina DE environment is different than in a classroom, and presumably experience in doing it shouldproduce better achievement results in students. However, this was not the case. Surprisingly,though, instructors’ experience with technology did make a difference (g = +0.1055, k = 23, p .05). By contrast, when students wereencouraged to communicate with their instructor, the result was positive and significant(+0.3079, k = 11, p < .05). There are several possibilities for this difference. First,encouragement does not necessarily produce the desired behavior, unless a structured, gradedassignment (e.g., collaborative problem-solving) brings students into contact with one another.The differences in these results may also be explained in terms of the more traditionalcommunication role of teacher and student and the less traditional communication role of studentand student (i.e., students tend to look to their teacher for help more than their peers). However,it is also possible that equal contact and communication resulted from this encouragement, butthat the teacher’s input was considered more salient or provided more assistance, especially asthis communication might relate to course objectives, testing, feedback or encouragement. Theremay be ways to explore the reasons behind this apparent contradiction which will be pursued asthe meta-analysis continues to develop.Surprisingly, two classes of learners appear not to have profited from DE: undergraduatecollege/university students and professional groups, like doctors and other health care workers.The finding for undergraduates is especially interesting because it is based on a large number ofoutcomes (k = 153). There may be factors, yet explored, that moderate these results. As well,military training applications appear to benefit from DE. This seems like a contradiction whenthis finding is contrasted with the positive findings for military personnel.

Meta-Analysis of the Comparative Literature in Distance Education 25What study features appear to detract from the effectiveness of DE compared to traditionalinstruction? Interestingly, there are only a few. When e-mail was used more in DE than theControl, a significantly negative effect occurred (g = –0.2937, k = 9, p < .05)—just the oppositeof what one would expect. E-mail should enhance DE, not hurt it. This result is counter-intuitiveand does not match the findings of Cavanaugh, with K-12 students. However, with such a smallsample size this finding is tenuous at best.Courses related to business management (e.g., economics, accounting, management, businesscommunication) were adversely affected by DE (g = –0.1113, k = 41, p < .05). Like the findingfor military training courses and military personnel, this seems to contradict the positive findingfor business and industry employees. Again closer examination of these seemingly anomalousfindings may uncover interesting relationships.Retention and Attitude OutcomesIn terms of other outcomes, it should come as no surprise that retention rate (i.e., the opposite ofattrition) was significantly higher in face-to-face classrooms than in DE applications. Thisfinding has been documented repeatedly over the history of DE research. Past researchers havemused that this situation, often attributed to isolation and lack of timely feedback andencouragement, would abate as communication technologies, such as e-mail and CMC, madeaccess to an instructor and a student’s peers more timely, transparent and fluid. There is evidencehere that it has not, although it is difficult to compare these results with the 50% attrition rate thathas been sometimes attributed to DE. One factor that could influence the “dropout rate” in DEand might partially explain the continuation of this phenomenon is “students’ experience inprevious DE courses.” It is possible that dropout is higher among inexperienced DE studentsthan experienced students. However, the small number of studies reporting previous experiencewith DE (k = 2) makes a determination of this sort impossible.The individual attitude results yield a more complex problem of interpretation. Possibly the mostreliable and telling result, “attitude towards course,” (k = 66, N = 9,097) produced an averageeffect size of –0.0103, which is consistent with the findings of Allen, Bourhis Burrell and Mabry(2002) for measures of “course satisfaction” with 23 outcomes and about 4,000 students. Thelarger negative results (attitudes towards “subject matter” and “instructor”) are both based onfewer effect sizes (k = 11 and 29, respectively) and therefore are likely to be more unstable. Thefinding for “attitude towards instructor” is perhaps interpretable in light of the relative anonymityof many DE teachers (compared to classroom teachers), especially in courses that rely solelyupon text-based communication (e.g., e-mail). In such instances, the “personality of theinstructor” is likely to be more neutral and the bonds that can be established between teacher andstudent in face-to-face instruction less dominant. The significantly positive finding of “attitudestowards technology,” based on 56 cases, may have resulted from the reliance on and perhaps thenovelty of technology in DE.One of the most disappointing aspects of this analysis is the difficulty that has been experiencedin separating the media of teaching from the media of learning (Keegan’s distinction between“distance teaching” and distance learning”). It was Kozma and Cobb who surmised that thebalance of Clark’s conclusion concerning the effects of media might shift as media more usefullyempowered students to engage in deeper learning processes to achieve learning outcomes that gobeyond those of retention and comprehension. However, largely because of the lack of

Meta-Analysis of the Comparative Literature in Distance Education 26sophistication in research design, measurement and reporting, it has been difficult to drawconclusions regarding the possibility of such a shift.Directions for Future DE ResearchWhat does this analysis suggest about future DE research directions? The answer to this questiondepends, somewhat, upon whether one accepts the premise of Clark and others, that mediacomparison studies (and DE comparison studies, by extension) answer few useful questions, orthe premise of Smith and Dillon, that there is still a place for comparative studies, performedunder certain conditions. It is probably true that, once established as a “legitimate alternative toclassroom instruction,” the need for comparative DE studies diminishes. After all, even in theworld of folk lore, the comparison between a steam-driven device and the brawn of John Henrywas performed only once, to the demise of the man. But it is also true that before we forge aheadinto an indeterminate future, possibly embracing untested fads and following false leads, while atthe same time dismantling the infrastructure of the past, we should reflect upon why we aregoing there and what we risk if we are wrong. And if there is a practical way of translating whatwe know about “best practices in the classroom” to “best practices in cyberspace,” then a casefor continued research in both venues, simultaneously, might be made.So what is it that we can learn from classroom instruction that can be translated into effective DEpractices? One of the few significant findings that emerged from the TV studies of the 50s and60s was that planning and design pay off—it was not the medium that mattered so much as whatcame before the TV cameras were turned on. Similarly, in this millennium, we might ask if thereare aspects of design, relating to either medium or method, that are optimal in either or bothinstructional contexts. In the review of this literature, few factorial designs were encountered,suggesting that the bulk of the studies ask questions in the form of, “Is it this or that?” Morecomplex designs might enable us to address questions such as, “What does it depend on?” Aspreviously mentioned, the majority of the “media effects” findings involved conditions of twowaysynchronicity. However, synchronicity in some ways defeats the advantage of “any place,any time learning” that has been the hallmark of DE. For instance, videoconferencing requiresstudents to be in a given place, at a given time, in the same way that face-to-face instructiondoes. So what is the best mix of media and methods for asynchronous distance education? Theanswer to this question has not been revealed in this meta-analysis and remains largely unknown.Is There an Optimal DE Comparison Study?In the best of all Campbell and Stanley worlds, an experiment that intends to establish causeeliminates all rival hypotheses and varies only one aspect of the design—the treatment. Thismeans, among other things, randomly assigning participants to treatment groups, holding historyconstant and controlling for the maturation of research participants. Such luxuries are difficultunder the best of circumstances, even in regular classrooms where greater experimental control istheoretically possible, much less in DE situations where control is very difficult to establish (i.e.,largely because of the distance). And there is the thorny problem of the delicate balance betweeninternal and external validity. Gaining more control usually means reducing generalizations toreal conditions, but lack of control means reduced interpretability. There is also the problem ofthe equivalence of two very different instructional circumstances. Even if the same instructoruses the same materials, in exactly the same ways, for the same amount of time, there areinstructional strategies that are enabled or limited by the mere fact of mediation over differenttimes and different places. Add to this the fact that DE and classroom-based students usually

Meta-Analysis of the Comparative Literature in Distance Education 27“self-select” themselves into these conditions, making selection bias difficult to neutralize. Thismyriad of limitations suggests that there is little possibility of designing the “perfect” DEcomparison study. But for the purpose of meta-analysis, perfection does not necessarily meantotal control; instead it means perfect description and reporting. Conditions that render nullhypothesis testing problematic, in order to establish cause in a single study, can still contribute toan overall consideration of the state of evidence in the population. So if studies of this sort are tocontinue, researchers must of course strive to reduce confounds and non-equivalent conditions,but they must also adopt an approach to accounting for their decision-making and researchpractices that makes their work available for future inspection and consideration.The Strengths and Limitations of Meta-AnalysisThe strengths and limitations of meta-analysis as an approach to investigating large quantitativeresearch literatures have been widely discussed and debated (e.g., Bernard & Naidu, 1990). Insummary, the major strengths are the following: 1) it is a systematic and comprehensiveapproach to investigation; 2) it answers questions of “how much” through the use of standardizedmetrics that can be combined; 3) it looks to explain variability in effect sizes; and 4) it allows themodeling of the “best subset” of factors that explain variability. The limitations of meta-analysisare as follows: 1) it does not allow for causal conclusions; 2) it excludes qualitative research andsingle case study research; and 3) it relies on the methodological quality, comprehensiveness and“popularity” of study features. The overall approach, methodology and attention to the details ofdata collection, coding and analysis demonstrate that this meta-analysis is a systematic andcomprehensive synthesis of the comparative literature of DE. However, the results should beinterpreted carefully in light of the general limitations of the meta-analytic approach. Resultsshould not be interpreted as representing causal connections, even though significance tests wereperformed on outcomes and study features. In addition, there is a body of qualitative research,case study and descriptive literature in distance education that has not been addressed here. Theresults reported here should also interpreted in light of both the quality and depth of the DEliterature itself. High variability among effect sizes, the quality of outcome measures,methodological confounds and the incompleteness of the literature are all factors that should beconsidered as one applies these findings to practice or plans future research efforts.The Future of this Meta-AnalysisThis meta-analysis is not complete. Further work will continue into the Summer of 2003.Thereare a number of recent “includes” that must be coded and one-third of the study features must bedouble-coded by a second coder. Beyond that, a reconsideration of codable study features andtheir levels that may be in order. Richard E. Clark (personal communication, February 4, 2003)has suggested some additional routes for that may be worth pursuing, if sufficient data areavailable in the literature. Adjustments to the codebook may result from this. In addition,multiple regression modeling and within-class post hoc comparisons are planned in an attempt tosort variables and combinations of variables in terms of their power to predict effect sizeoutcomes.ConclusionThis meta-analysis represents a rigorously applied examination of the comparative literature ofdistance education, with regard to the variety of conditions of study features and outcomes thatare publicly available. Overall, DE and traditional face-to-face instruction affect student learningsimilarly but the variability in outcomes is noteworthy. Conditions that contribute to more

Meta-Analysis of the Comparative Literature in Distance Education 28effective distance education include the use of synchronous communication and interactivelearning technologies such as computer mediated communication and two-way audio and twowayvideo. The results should be interpreted, however, in light of the incompleteness of the DEliterature.References* indicates a study that was included in the meta-analysis.Abrami, P.C., & Bures, E.M. (1996). Computer-supported collaborative learning and distanceeducation. American Journal of Distance Education, 10(2), 37-42.Abrami, P.C., Cohen, P., & d'Apollonia, S. (1988). Implementation problems in meta-analysis.Review of Educational Research, 58(2), 151-179.Allen, M., Bourhis, J., Burrell, N., & Mabry, E. (2002). Comparing student satisfaction withdistance education to traditional classrooms in higher education: A meta-analysis. TheAmerican Journal of Distance Education, 16(2), 83-97.*Anderson, M. R. (1993). Success in distance education courses versus traditional classroomeducation courses. Unpublished doctoral dissertation, Oregon State University.Anglin, G., & Morrison, G. (2000). An analysis of distance education research: Implications forthe instructional technologist. Quarterly Review of Distance Education, 1(3), 180-194.*Appleton, A. S., and others. (1989). Improved teaching excellence by using tutored videoinstruction: An Australian case study. Paper presented at the EAIR Forum,Trier,Germany.*Armstrong-Stassen, M., Landstorm, M., & Lumpkin, R. (1998). Student's reactions to theintroduction of videoconferencing for classroom instruction. Information Society, 14,153-164.*Bacon, S. F., & Jakovich, J. A. (2001). Instructional television versus traditional teaching of anintroductory psychology course. Teaching of Psychology, 28(2), 88-91.*Bader, M. B., & Roy, S. (1999). Using technology to enhance relationships in interactivetelevision classrooms. Journal of Education for Business, 74(6), 357-362.*Barker, B. M. (1994, September). Collegiate aviation review. Auburn, AL: University AviationAssociation.*Bartel, K. B. (1998). A comparison of students taught utilizing distance education andtraditional education environments in beginning microcomputer applications classes atUtah State University. Unpublished doctoral dissertation, Utah State University.

Meta-Analysis of the Comparative Literature in Distance Education 29*Bauer, J. W., & Rezabek, L. L. (1993). Effects of two-way visual contact on verbal interactionduring face-to-face and teleconferenced instruction. In Art, science & visual literacy:Selected readings from the annual conference of the International Visual LiteracyAssociation [ERIC Document Reproduction No. ED 363 299].*Beare, P. L. (1989). The comparative effectiveness of videotape, audiotape, and telelecture indelivering continuing teacher education. American Journal of Distance Education, 3(2),57-66.Berge, Z. L., & Mrozowski, S. (2001). Review of research in distance education, 1990 to 1999.American Journal of Distance Education, 15(3), 15-19.Bernard, R. M. (1986). Is research in new technology caught in the same old trap? CanadianJournal of Educational Communication, 15(3), 147-151.Bernard, R.M., & Amundsen, C.L. (1989). Antecedents to drop-out in distance education: Doesone model fit all? Journal of Distance Education, 4(2), 25-46.Bernard, R.M., & Naidu, S. (1990). Integrating research into practice: The use and abuse ofmeta-analysis. Canadian Journal of Educational Communication, 19(3), 171-195.*Bischoff, W. R., BIsconer, S. W., Kooker, B. M., & Woods, L. C. (1996). Transactionaldistance and interactive television in the distance education of health professionals.American Journal of Distance Education, 10(3), 4-19.*Boulet, M. M., Boudreault, S., & Guerette, L. (1998). Effects of a television distance educationcourse in computer science. British Journal of Educational Technology, 29(2), 101-111.*Britton, O. L. (1992). Interactive distance education in higher education and the impact ofdelivery styles on student perceptions. Unpublished doctoral dissertation, Wayne StateUniversity.*Browning, J. B. (1999). Analysis of concepts and skills acquisition differences between webdeliveredand classroom-delivered undergraduate instructional technology courses.Unpublished doctoral dissertation, North Carolina State University.*Bruning, R., Landis, M., Hoffman, E., & Grosskopf, K. (1993). Perspectives on an interactivesatellite-based Japanese language course. American Journal of Distance Education, 7(3),22-38.*Burkman, T. A. (1994). An analysis of the relationship of achievement, attitude, andsociological element of individual learning style of students in an interactive televisioncourse. Unpublished doctoral dissertation, Western Michigan University, Kalamazoo,Michigan.Campbell, D., & Stanley, J. (1963). Experimental and quasi-experimental designs for research.New York: Houghton Mifflin.

Meta-Analysis of the Comparative Literature in Distance Education 30*Card, K. A., & Horton, L. (1998, November 5-8). Fostering collaborative learning amongstudents taking higher education administrative courses using computer-mediatedcommunication. ASHE annual meeting paper. Paper presented at the annual meeting ofthe Association for the Study of Higher Education [ERIC document ReproductionService No. ED 427 612]. Miami, FL.*Carl, D. R., & Densmore, B. (1988). Introductory accounting on distance university educationvia television (duet): A comparative evaluation. Canadian Journal of EducationalCommunication, 17(2), 81-94.*Carrell, L. J., & Menzel, K. E. (2001). Variations in learning, motivation, and perceivedimmediacy between live and distance education classrooms. Communication Education,50(3), 230-240.Carter, V. (1996, February). Do media influence learning? Revisiting the debate in the context ofdistance education. Open Learning, 31-40.Cavanaugh, C.S. (2001). The effectiveness of interactive distance education technologies in K-12 learning: A meta-analysis. International Journal of Educational Telecommunications,7(1), 73-88. Available online: http://www.unf.edu/~ccavanau/CavanaughIJET01.pdf*Chapman, A. D. (1996). Using interactive video to teach learning theory to undergraduates:Problems and benefits. [ERIC Document Reproduction Service ED 406 425].*Cheng, H. C., Lehman, & Armstrong, P. (1991). Comparison of performance and attitude intraditional and computer conferencing classes. American Journal of Distance Education,5(3), 51-64.*Cho, E. (1998). Analysis of teacher & students' attitudes on two-way video tele-educationalsystem for Korean elementary school. Educational Technology Research andDevelopment, 46(1), 98-105.*Chung, J. (1991). Televised teaching effectiveness: Two case studies. Educational Technology,31 41-47.*Chute, A. G., Balthazar, L. B., & Posten, C. O. (1988). Learning from tele-training. AmericanJournal of Distance Education, 2(3), 55-63.*Chute, A. G., Hulik, M., & Palmer, C. (1987, May 5). Teletraining productivity at AT&T. Paperpresented at International Teleconferencing Association annual convention, Washington,D.C.*Clark, B. A. (1989, March 12). Comparisons of achievement of students in on-campusclassroom instruction versus satellite teleconference instruction. Paper presented at thenational conference on Teaching Public Administration [ERIC Document ReproductionService No. ED 327 668], Charlottesville, VA.Clark, R.E. (1983). Reconsidering research on learning from media. Review of EducationalResearch. 53(4), 445-459.

Meta-Analysis of the Comparative Literature in Distance Education 31Clark, R.E. (1994). Media will never influence learning. Educational Technology Research andDevelopment, 42(2), 21-29.Clark, R.E. (2000). Evaluating distance education: Strategies and cautions. Quarterly Review ofDistance Education, 1(1), 3-16.*Coates, D., Humphreys, B. R., Kane, J., Vachris, M., Agarwal, R., & Day, E. (2001, January 5-7). "No significant distance" between face to face and online instruction: Evidence fromprinciples of economics. Paper presented at the meeting of the Allied Social ScienceAssociation . New Orleans, LA.Cobb, T. (1997). Cognitive efficiency: Toward a revised theory of media. EducationalTechnology Research & Development, 45(4), 21-35.*Collins, M. (2000). Comparing Web correspondence and lecture versions of a second-year nonmajorbiology course. British Journal of Educational Technology, 31(1), 21-27.*Cooper, L. W. (2001). A comparison of online and traditional computer applications classes.T.H.E. Journal, 28(8), 52-58.*Cordover, P. P. (1996). A comparison of a distance education and locally based course in anurban university setting. Unpublished doctoral dissertation, Florida InternationalUniversity.*Cross, R. F. (1996). Video-taped lectures for honours students on international industry basedlearning. Distance Education, 17(2), 369-386.*Curnow, C. K. (2001). Social interaction, learning styles, and training outcomes: Differencesbetween distance learning and traditional training. Unpublished doctoral dissertation,George Washington University.*Davis, J. L. (1996). Computer-assisted distance learning, part II: Examination performance ofstudents on & off campus. Journal of Engineering Education,(January issue), 77-82.*Day, T. M., Raven, M. R., & Newman, M. E. (1997). The effects of World Wide Webinstructional and traditional instruction and learning styles on achievement and changesin student attitudes in a technical writing in agricommunication course. Paper presentedat the annual meeting of the National Agricultural Education Research Association, LasVegas, Nevada.*Day, T. M., Raven, M. R., & Newman, M. E. (1998). The effects of world wide web instructionand traditional instruction and learning styles on achievement and changes in studentattitudes in a technical writing in agricommunication course. Journal of AgriculturalEducation, 39(4), 65-75.*Dees, S. C. (1994). An investigation of distance education versus traditional course deliveryusing comparisons of student achievement scores in advanced placement chemistry andperceptions of teachers and students about their delivery system. Unpublished doctoraldissertation, Northern Illinois University.

Meta-Analysis of the Comparative Literature in Distance Education 32*Dexter, D. J. (1995). Student performance-based outcomes of televised interactive communitycollege . Unpublished doctoral dissertation, Colorado State University.Diaz, D.P. [Online]. Carving a new path for distance education research. The Technology Source.March/April. Available at: http://horizon.unc.edu/TS/default.asp?show=articleandid=68*DiBartola, L. M., Miller, M. K., & Turley, C. L. (2001). Do learning style and learningenvironment affect learning outcome? Journal of Allied Health, 30(2), 112-115.*Dillon, C. L., Gunawardena, C. N., & Parker, R. (1992). Learner support: The critical link indistance education. Distance Education, 13(1), 29-45.*Dominiguez, P. S., & Ridley, D. R. (2001). Assessing distance education courses and disciplinedifferences in their effectiveness. Journal of Instructional Psychology, 28(1), 15-19.*Faux, T. L., & Black-Hughes, C. (2000). A comparison of using the Internet versus lectures toteach social work history. Research on Social Work Practice, 10(4), 454-466.*Foell, N. A. (1989, December). Using computers to provide distance learning, the newtechnology. Paper presented at the annual meeting of the American Vocational EducationResearch Association [ERIC Document Reproduction Service No. ED 316 714],Orlando, FL.*Freitas, F. A., Myers, S. A., & Avtgis, T. A. (1998). Student perceptions of instructorimmediacy in conventional and distributed learning classrooms. CommunicationEducation, 47(4), 366-372.*Fritz, S., Bek, T. J., & Hall, D. L. (2001). Comparison of campus and distance undergraduateleadership students' attitudes. Journal of Behavioural and Applied Management, 3(1), 3-12.*Furste-Bowe, J. A. (1997). Comparison of student reactions in traditional andvideoconferencing courses in training and development. International Journal ofInstructional Media, 24(3), 197-205.*Fyock, J. J. (1994). Effectiveness of distance learning in three rural schools as perceived bystudents (student-rated). Unpublished doctoral dissertation, Cornell University.Garrison, D.R., Anderson, T. & Archer, W. (2001). Critical thinking, cognitive presence, andcomputer conferencing in distance education. American Journal of Distance Education,15(1), 7-23.Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. BeverlyHills, CA: Sage.*Glenn, A. S. (2001). A comparison of distance learning and traditional learning environments.[ERIC Document Reproduction Service No. ED 457 778].

Meta-Analysis of the Comparative Literature in Distance Education 33*Goodyear, J. M. (1995). A comparison of adult students' grades in traditional and distanceeducation courses. Unpublished doctoral dissertation, University of Alaska.*Gray, B. A. (1996). Student achievement and temperament types in traditional and distancelearning environments (urban education, traditional education). Unpublished doctoraldissertation, Wayne State University.*Grimes, P. W., Neilsen, J. E., & Ness, J. F. (1988). The performance of nonresident students inthe "economics u$a" telecourse. American Journal of Distance Education, 2(2), 36-43.*Hackman, M. Z., & Walker, K. B. (1994, July 11-15). Perceptions of proximate and distantlearners enrolled in university-level communication courses: a significant nonsignificantfinding. Paper presented at the 44th annual meeting of the International CommunicationAssociation [ERIC Document Reproduction Service No. ED 372 428]. Sydney, NewSouth Wales, Australia.*Hahn, H. A., & et al. (1990). Distributed training for the reserve component: Remote deliveryusing asynchronous computer conferencing. Idaho, Falls: U.S. Army Research Institutefor the Behavioral and Social Sciences (Report No 1581) [ERIC Document ReproductionService No. ED 359 918].Hawkes, M. (2001). Variables of interest in exploring the reflective outcomes of network-basedcommunication. Journal of Research on Computing in Education, 33(3) 299-315.Hedges, L.V. & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: AcademicPress.Hedges, L.V., Shymansky, J.A., & Woodworth, G. (1989). A practical guide to modernmethods of meta-analysis. [ERIC Document Reproduction Service No. ED 309952].*Heiens, R. A., & Hulse, D. B. (1996). Two-way interactive television: An emerging technologyfor university level business school instruction. Journal of Education for Business, 72(2),74-77.*Hiltz, S. R. (1993). Correlates of learning in a virtual classroom. International Journal of ManMachine Studies, 39(1), 71-98.*Hinnant, E. C. (1994). Distance learning using digital fiber optics: A study of studentachievement and student perception of delivery system quality. Unpublished doctoraldissertation, Mississippi State University.*Hittelman, M. (2001). Distance education report: Fiscal years 1995-1996 through 1999-2000.Sacramento, CA: California Community Colleges, Office of the Chancellor.

Meta-Analysis of the Comparative Literature in Distance Education 34*Hodge-Hardin, S. (1997). Interactive television vs. a traditional classroom setting: Acomparison of student math achievement. In Mid-South Instructional TechnologyConference Proceedings [ERIC document Reproduction Service No. ED 430 521].Murfreesboro, TN.*Hoey, J. J., Pettitt, J. M., Brawner, C. E., & Mull, S. P. (1998). Project 25: First semesterassessment- A report on the implementation of courses offered on the Internet. RetrievedMarch,1997 from http://www2.ncsu.edu/ncsu/ltc/Project25/info/f97_assessment.html*Hogan, R. (1997, July 23-25). Analysis of student success in distance learning coursescompared to traditional courses. Paper presented at the 6th annual conference onMultimedia in Education and Industry [Eric Document Reproduction no. ED 412 992].Beaufort, SC: Technical College of the Low Country.*Huff, M. T. (2000). A comparison study of live instruction versus interactive television forteaching MSW students critical thinking skills. Research on Social Work Practice, 10(4),400-416.*Hurlburt, R. T. (2001). "Lectlets" delivery content at a distance: Introductory statistics as a casestudy. Teaching of Psychology, 28(1), 15-20.*Jeannette, K. J., & Meyer, M. H. (2002). Online learning equals traditional classroom trainingfor master gardeners. HortTechnology, 12(1), 148-156.*Jewett, F. (1998). The education network of Maine: A case study in the benefits and costs ofinstructional television. Seal Beach. CA: California State University, Office of theChancellor.*Jewett, F. (1998). The Westnet program--Suny Brockport and the Suny campuses in WesternNew York state: A case study in the benefits and costs of an interactive televisionnetwork. [ERIC Document Reproduction Service No. ED 420 301].*Johnson, G. R., O'Connor, M., & Rossing, R. (1985). Interactive two-way television: Revisited.Journal of Educational Technology Systems, 13(3), 153-158.*Johnson, K. R. (1993). An analysis of variables associated with student achievement andsatisfaction in a university distance education course. Unpublished doctoral dissertation,New York State University, Buffalo.*Johnson, M. Introductory biology online: Assessing outcomes of two student populations.Journal of College Science Teaching, 31(5), 312-317.*Johnson, S. D., Aragon, S. R., Shaik, N., & Palma-Rivas, N. (1999). Comparative analysis ofonline vs. face-to-face instruction. Champaign, IL: Department of Human ResourceEducation, University of Illinois at Urbana-Champaign.

Meta-Analysis of the Comparative Literature in Distance Education 35*Johnson, S. M. (2001). Teaching introductory international relations in an entirely web-basedenvironment: comparing student performance across and within groups. Ed at a Distance,15(10).*Jones, E. R. (1999, February 28 - March 4). A comparison of an all web-based class to atraditional class. Paper presented at the Society for Information Technology & TeacherEducation International conference [ERIC document Reproduction Service No. ED 432286]. San Antonio, TX.Jung, I., & Rha, I. (2000, July-August). Effectiveness and cost-effectiveness of onlineeducation: A review of the literature. Educational Technology, 57-60.*Kabat, E. J., & Friedel, J. N. (1990). The Eastern Iowa community college districts televisedinteractive education evaluation report. Clinton, IA: Eastern Iowa Community College.*Kahl, T. N., & Cropley, A. J. (1986). Face-to-face versus distance learning: Psychologicalconsequences and practical implications. Distance Education, 7(1), 38-48.*Kataoka, H. C. (1987). Long-distance language learning: The second year of televisedJapanese. Journal of Educational Techniques and Technologies, 20(2), 43-50.Keegan, D. (1996). Definition of distance education. In Foundations of distance education. (3 rded). London: Routledge.*Keene, S. D., & Cary, J. S. (1990). Effectiveness of distance education approach to U.S. ArmyReserve component training. American Journal of Distance Education, 4(2), 14-20.*Kennedy, R. L., Suter, W. N., & Clowers, R. L. (1997, November 12-14). Research byelectronic mail. Paper presented at the Annual Meeting of the Mid-South EducationalResearch Association [ERIC Document Reproduction Service no. ED 415 274].Memphis, TN.*King, F. B. (2001). Asynchronous distance education courses employing web-based instruction:Implications of individual study skills self-efficacy and self-regulated learning.Unpublished doctoral dissertation, University of Connecticut.*Kochman, A., & Maddux, C. D. (2001). Interactive televised distance learning versus oncampusinstruction: A comparison of final grades. Journal of Research on Technology inEducation, 34(1), 87-91.Kozma, R.B. (1994). Will media influence learning? Reframing the debate. EducationalTechnology Research & Development, 42(2), 7-19.*Kranc, B. M. (1997). The impact of individual characteristics on telecommunication distancelearning cognitive outcomes in adult/nontraditional students. Unpublished doctoraldissertation, North Carolina State University.

Meta-Analysis of the Comparative Literature in Distance Education 36*Kretovics, M. A. (1998). Outcomes assessment: The impact of delivery methodologies andpersonality preference on student learning outcomes. Unpublished doctoral dissertation,Colorado State University.*Larson, M. R., & Bruning, R. (1996). Participant perceptions of a collaborative satellite-basedmathematics course. American Journal of Distance Education, 10(1), 6-22.*Lia-Hoagberg, B., Vellenga, B., Miller, M., & Li, T. Y. (1999). A partnership model of distanceeducation: Students' perceptions of connectedness and professionalization. Journal ofProfessional Nursing, 15(2), 116-122.*Liang, C. C. (2001). Guidelines for distance education: A case study in Taiwan. Journal ofComputer Assisted Learning, 17(1), 48-57.*Litchfields, R. E., Oakland, M. J., & Anderson, J. A. (2002). Relationships between interncharacteristics, computer attitudes, and use of online instruction in a dietetic trainingprogram. American Journal of Distance Education, 16(1), 23-36.Lou, Y. & MacGregor, S. K. (under review). Enhancing project-based learning through onlinebetween-group collaboration. Educational Research and Evaluation [Special Issue onPostsecondary Education].Lou, Y., Dedic, H., & Rosenfield, S. (2003). Feedback model and successful e-learning. In SomNaidu (Ed.), Learning and teaching with technology: Principles and practice (pp. 249-260). London and Sterling, VA: Kogan Page.*MacFarland, T. W. (1998). A comparison of final grades in courses when faculty concurrentlytaught the same course to campus-based and distance education students: winter term1997. Fort Lauderdale, FL: Nova Southeastern University.*MacFarland, T. W. (1999). Matriculation status of fall term 1993 center for psychologicalstudies students by the beginning of fall term 1998: Campus-based students and distanceeducation students by site. [ERIC Document Reproduction Service No. ED 434 557].Machtmes, K. & Asher, J.W. (2000). A meta-analysis of the effectiveness of telecourses indistance education. American Journal of Distance Education. 14(1): 27-46.McKnight, C.B. (2001). Supporting critical thinking in interactive learning environments.Computers in the Schools, 17(3-4), 17-32.*Magiera, F. T. (1994). Teaching managerial finance through compressed video: An alternativefor distance education. Journal of Education for Business, 69(5), 273-277.*Magiera, F. T. (1994-1995). Teaching personal investments via long-distance. Journal ofEducational Technology Systems, 23(4),*Maki, R. H., Maki, W. S., Patterson, M., & Whittaker, P. D. (2000). Evaluation of a Web-basedintroductory psychology course: I. Learning and satisfaction in on-line versus lecturecourses. Behaviour Research Methods, Instruments, & Computers, 32, 230-239.

Meta-Analysis of the Comparative Literature in Distance Education 37*Maki, W. S., & Maki, R. H. (2002). Multimedia comprehension skill predicts differentialoutcomes of Web-based and lecture courses. Journal of Experimental Psychology:Applied, 8(2), 85-98.*Martin, E. D., & Rainey, L. (1993). Student achievement and attitude in a satellite-deliveredhigh school science course. American Journal of Distance Education, 7(1), 54-*Maxcy, D. O., & Maxcy, S. J. (1986-1987). Computer/telephone pairing for long distancelearning. Journal of Educational Technology Systems, 15(2), 201-211.*McCleary, I. D., & Egan, M. W. (1989). Program design and evaluation: Two-way interactivetelevision. American Journal of Distance Education, 3(1), 50-60.*McGreal, R. (1994). Comparison of the attitudes of learners taking audiographicteleconferencing courses in secondary schools in Northern Ontario. InterpersonalComputing and Technology Journal, 2(4), 11-23.*McKissack, C. E. (1997). A comparative study of grade point average (GPA) between thestudents in traditional classroom setting and the distance learning classroom setting inselected colleges and universities. Unpublished doctoral dissertation, Tennessee StateUniversity.*Miller, J. W., McKenna, M. C., & Ramsey, P. (1993). An evaluation of student content learningand affective perceptions of a two-way interactive video learning experience.Educational Technology, 33(6), 51-55.*Mills, B. D. (1998). Comparing optimism and pessimism of students in distance-learning andon campus. Psychological Reports, 83(3, Pt 2), 1425-1426.*Mills, B. D. (1998). Replication of optimism and pessimism of distance-learning and oncampusstudents. Psychological Reports, 83(3, Pt 2), 1454+.Moore, M., & Thompson, M. (1990). The effects of distance learning: A summary of theliterature. University Park, PA: The Pennsylvania State University. [ERIC DocumentReproduction Service No. ED 391 467].*Moorhouse, D. R. (2001). Effect of instructional delivery method on student achievement in amaster's of business administration course at the Wayne Huizenga School of Businessand Entrepreneurship. Ft. Lauderdale, FL: Nova Southeastern University.Morrison, G. R. (1994). The media effects question: “Unresolveable” or asking the rightquestion. Educational Technology Research and Development, 42(2), 41-44.*Moshinskie, J. F. (1997). The effects of using constructivist learning models when deliveringelectronic distance education (EDE) courses: A perspective study. Journal of InstructionDelivery Systems, 11(1), 14-20.*Murray, J. D., & Heil, M. (1987). Project evaluation: 1986-87 Pennsylvania teleteachingproject. Mansfield, PA: Mansfield University of Pennsylvania.

Meta-Analysis of the Comparative Literature in Distance Education 38*Muta, H., Kikuta, R., Hamano, T., & Maesako, T. (1997). The effectiveness of low-cost telelecturing.Staff and Educational Development International, 1(2-3), 129-142.*Mylona, Z. H. (1999). Factors affecting enrollment satisfaction and persistence in a Web-basedvideo-based and conventional instruction. Unpublished doctoral dissertation, Universityof Southern California.*Nakshabandi, A. A. (1993). A comparative evaluation of a distant education course for femalestudents at King Saud University. International Journal of Instructional Media, 20(2),127-136.Nipper, S. (1989). Third generation distance learning and computer conferencing. In: R. Masonand A. Kaye (Eds), Mindweave: Communication, computers and distance education (pp.63-73). Oxford, UK: Pergamon Press.*Obermier, T. R. (1991). Academic performance of video-based distance education students andon-campus students. Unpublished doctoral dissertation, Colorado State University.*Olejnik, S., & Wang, L. (1992-1993). An innovative application of the macintosh classic iicomputer for distance education. Journal of Educational Technology Systems, 21(2), 87-101.*Parkinson, C. F., & Parkinson, S. B. (1989). A comparative study between interactive televisionand traditional lecture course offerings for nursing students. Nursing and Health Care,10(9), 498-502.Perraton, H. (2000). [Online]Rethinking the research agenda. International Review of ResearchinOpen and Distance Learning. 1(1). Available at: http://www.irrodl.org/v1.1html*Petracchi, H. E., & Patchner, M. A. (2000). Social work students and their learningenvironment: A comparison of interactive television, face-to-face instruction, and thetraditional classroom. Journal of Social Work Education, 36(2), 335-346.*Petracchi, H. E., & Patchner, M. E. (2001). A comparison of live instruction and interactivetelevised teaching: A 2-year assessment of teaching an MSW research methods course.Research on Social Work Practice, 11(1), 108-117.*Phelps, R. H., Wells, R. A., Ashworth, R. L., & Hahn, H. A. (1991). Effectiveness and costs ofdistance education using computer-mediated communication. American Journal ofDistance Education, 5(3), 7-19.*Phillips, M. R., & Peters, M. J. (1999). Targeting rural students with distance learning courses:A comparative study of determinant attributes and satisfaction levels. Journal ofEducation for Business, 74(6), 351-356.Phipps, R. & Merisotis, J. (1999). What’s the difference? A review of contemporaryresearch on the effectiveness of distance learning in higher education.Washington, DC: The Institute for Higher Education Policy.

Meta-Analysis of the Comparative Literature in Distance Education 39*Pirrong, G. D., & Lathen, W. C. (1990). The use of interactive television in business education.Educational Technology, 30 (May) 49-54.*Pugh, R. C., & Siantz, J. E. (1995, April). Factors associated with student satisfaction indistance education using slow scan television. Paper presented at the annual meeting ofthe American Educational Research Association. San Francisco.*Richards, I. E. (1994). Distance learning: A study of computer modem students in a communitycollege. Unpublished doctoral dissertation, Kent State University.*Richards, I., & and others. (1995, April 18-22). A study of computer-modem students: A call foraction. Paper presented at the annual meeting of the American Educational ResearchAssociation [ERIC Document Reproduction Service No. ED 391 467] San Francisco.*Ritchie, H., & Newby, T. J. (1989). Classroom lecture/discussion vs. live televised instruction:A comparison of effects on student performance, attitudes, & interaction. The AmericanJournal of Distance Education, 3(3), 36-45.*Rost, R. C. (1997). A study of the effectiveness of using distance education to present trainingprograms to extension service master gardener trainees. Unpublished doctoraldissertation, Oregon State University.*Rudin, J. P. (1998). Teaching undergraduate business management courses on campus and inprisons. Journal of Correctional Education, 49(3), 100-106.*Rudolph, S., & Gardner, M. K. (1986-1987). Remote site instruction in physics: A test of theeffectiveness of a new teaching technology. Journal of Educational Technology Systems,15(1), 61-80.Russell, T.L. (1999). The no significant difference phenomenon. Chapel Hill, NC: Officeof Instructional Telecommunications, North Carolina State University.*Ryan, W. F. (1996). The distance education delivery of senior high advanced mathematicscourses in the province of Newfoundland and Labrador: A study of the academicprogress of the participating students. Unpublished doctoral dissertation, OhioUniversity.*Ryan, W. F. (1996). The effectiveness of traditional vs. audiographics delivery in senior highadvanced mathematics courses. Journal of Distance Education, 11(2), 45-55.Saba, F. (2000). Research in distance education: A status report. International Review ofResearch in Open and Distance Education, 1(1), 1-9.*Sankar, C. S., Ford, F. N., & Terase, N. (1998). Impact of videoconferencing in teaching anintroductory MIS course. Journal of Educational Technology Systems, 26(1), 67-85.*Sankaran, S. R., & Bui, T. (2001). Impact of learning strategies and motivation on performance:A study in Web-based instruction. Journal of Instructional Psychology, 28(3), 191-198.

Meta-Analysis of the Comparative Literature in Distance Education 40*Sankaran, S. R., Sankaran, D., & Bui, T. X. (2000). Effect of student attitude to course formaton learning performance: An empirical study in Web vs. lecture instruction. Journal ofInstructional Psychology, 27(1), 66-73.Schlosser, C.A., & Anderson, M.L. (1994). Distance education: Review of the literature.Washington, DC: Association for Educational Communications and Technology.*Schoenfeld Tacher, R., & McConnell, S. (2001, April 10-14). An examination of the outcomesof a distance-delivered science course. Paper presented at the annual meeting of theAmerican Educational Research Association. Seattle, WA.*Schoenfeld Tacher, R., McConnell, S., & Graham, M. (2001). Do no harm--a comparison of theeffects of on-line vs. traditional delivery media on a science course. Journal of ScienceEducation and Technology, 10(3), 257-265.*Schutte, J. G. (1997).Virtual teaching in higher education: The new intellectual superhighwayor just another traffic jam? Retrieved November, 2000 fromhttp://www.csun.edu/sociology/virexp.htm*Scott, M. (1990). A comparison of achievement between college students attending traditionaland television course presentations(distance education). Unpublished doctoraldissertation, Auburn University.*Searcy, R. (1993). Grade distribution study: Telecourses vs. traditional courses. (Studyprepared for the Calhoum Community College Steering Committee). (ERIC DocumentReproduction Service No. ED 362 251).Shachar, M. (2002). Differences between traditional and distance education outcomes: A metaanalyticapproach. Unpublished dissertation, Touro University International.Shale, D. (1990). Toward a reconceptualization of distance education. In M. G. Moore(Ed.), Contemporary issues in American distance education (pp. 333-343).Oxford: Pergamon Press.*Simpson, H., Pugh, H. L., & Parchman, S. W. (1991). An experimental two-way videoteletraining system: Design, development and evaluation. Distance Education, 12(2),209-231.*Simpson, H., Pugh, H. L., & Parchman, S. W. (1993). Empirical comparison of alternativeinstructional TV technologies. Distance Education, 14(1), 147-164.*Sipusic, M. J., Pannoni, R. L., Smith, R. B., Dutra, J., Gibbons, J. F., & Sutherland, W. R.(1999). Virtual collaborative learning: A comparison between face-to-face tutored video(TVI) and distributed tutored video instruction (DTVI) (SML technical report series). SanFrancisco: Sun Microsystems.*Sisung, N. J. (1992). The effects of two modes of instructional delivery: Two-way forwardfacing interactive television and traditional classroom on attitudes, motivation, ontask/off-taskbehavior and final exam grades of students enrolled in humanities courses.

Meta-Analysis of the Comparative Literature in Distance Education 41Unpublished doctoral dissertation, University of Michigan.

Meta-Analysis of the Comparative Literature in Distance Education 42*Smith, D. L., & McNelis, M. J. (1993, April 12-16). Distance education: Graduate studentattitudes and academic performance. Paper presented at the annual meeting of theAmerican Educational Research Association [Eric Document Reproduction Service no.ED 360 948]. Atlanta, GA.Smith, P. L. & Dillon, C. L. (1999). Comparing distance learning and classroom learning:Conceptual considerations. American Journal of Distance Education, 13, 107-124.*Smith, R. E. (1990). Effectiveness of the interactive satellite method in the teaching of first yearGerman: A look at selected high schools in Arkansas and Mississippi. Unpublisheddoctoral dissertation, University of Mississippi.*Smith, T. E. (2001). A comparison of achievement between community college studentsattending traditional and video course presentations. Unpublished doctoral dissertation,Auburn University.*Sorensen, C. K. (1996). Students near and far: Differences in perceptions of community collegestudents taking interactive television classes at origination and remote sites. [ERICDocument Reproduction Service No. ED 393 509].*Souder, W. E. (1993). The effectiveness of traditional vs. satellite delivery in three managementof technology master's degree programs. American Journal of Distance Education, 7(1),37-53.*Spooner, F., Jordan, L., Algozzine, B., & Spooner, M. (1999). Student ratings of instruction indistance learning and on-campus classes. Journal of Educational Research, 92(3), 132-140.*Stone, H. R. (1990). Does interactivity matter in video-based off-campus graduate engineeringeducation? Washington, DC: American Society for Engineering Education, Center forProfessional Development. [ERIC Document Reproduction Service no. ED 317 421].Sumner, J. (2000). Serving the System: A critical history of distance education. Open Learning,15(3), 267-295.*Suter, N. W., & Perry, M. K. (1997, November 12-14). Evaluation by electronic mail. Paperpresented at the annual meeting of the Mid-South Educational Research Association [EricDocument Reproduction Service no. ED 415 269]. Memphis, TN.Taylor, J.C. (2001). Fifth generation distance education. Keynote address delivered at the ICDE20th World Conference, Dusseldorf, Germany, 1-5 April. Available on-line at:http://www.usq.edu.au/users/taylorj/conferences.htmTennyson, R.D. (1994). The big wrench vs. integrated approaches: The great media debate.Educational Technology, Research and Development, 42(2), 15-28.

Meta-Analysis of the Comparative Literature in Distance Education 43*Thomerson, J. D. (1995). Student perceptions of the affective experiences encountered indistance learning courses (interactive television). Unpublished doctoral dissertation,University of Georgia.*Tidewater Community College. (2001). Distance learning report. Norfolk, VA: TidewaterCommunity College, Office of Institutional Effectiveness.*Tiene, D. (1997). Student perspective on distance learning with interactive television.TechTrends , 42(1), 41-47.*Timmins, K. (1989). Educational effectiveness of various adjuncts to printed study material indistance education. Research in Distance Education, 1(3), 12-13.*Toussaint, D. (1990). Fleximode: Within Western Australia TAFE. Leabrook, Australia: TAFENational Centre for Research and Development. [ERIC Document Reproduction ServiceNo. ED 333 227].Ullmer, E.J. (1994). Media and learning: Are there two kinds of truth? Educational TechnologyResearch & Development, 42 (1), 21-32.*Umble, K. E., Cervero, R. M., Yang, B., & Atkinson, W. L. (2000). Effects of traditionalclassroom and distance continuing education: a theory-driven evaluation of a vaccinepreventablediseases course. American Journal of Public Health, 90(8), 1218-1224.*Waldmann, E., & De Lange, P. (1996). Performance of business undergraduates studyingthrough open learning: A comparative analysis. Accounting Education, 5(1), 25-33.*Walker, B. M., & Donaldson, J. F. (1989). Continuing engineering education by electronicblackboard and videotape: A comparison of on-campus and off-campus studentperformance. IEEE Transactions on Education, 32(4), 443-447.*Wallace, L. F., & Radjenovic, D. (1996).Remote training for school teachers of children withdiabetes mellitus. Retrieved September, 2001 fromhttp://www.unb.ca/naweb/proceedings/1996/zwallace.html*Wang, A. Y., & Newlin, M. H. (2000). Characteristics of students who enroll and succeed inpsychology web-based classes. Journal of Educational Psychology, 92(1), 137-143.*Waschull, S. B. (2001). The online delivery of psychology courses: Attrition, performance, andevaluation. Teaching of Psychology, 28(2), 143-146.*Wegner, S. B., Holloway, K. C., & Garton, E. M. (1999). The effects of Internet-basedinstruction on student learning. Journal of Asynchronous Learning Networks, 3 (2).*Westbrook, T. S. (1998). Changes in students' attitude toward graduate business instruction viainteractive television. American Journal of Distance Education, 11(1), 55-69.

Meta-Analysis of the Comparative Literature in Distance Education 44*Whetzel, D. L., Felker, D. B., & Williams, K. M. (1996). A real world comparison of theeffectiveness of satellite training and classroom training. Educational TechnologyResearch and Development, 44(3), 5-18.*Whitten, P., Ford, D. J., Davis, N., Speicher, R., & Collins, B. (1998). Comparison of face-tofaceversus interactive video continuing medical education delivery modalities. Journalof Continuing Education in the Health Professions, 18(2), 93-99.*Wick, W. R. (1997). An analysis of the effectiveness of distance learning at remote sites versuson-site locations in high school foreign language programs. Unpublished doctoraldissertation, University of Minnesota.Winkelmann, C.L. (1995). Electronic literacy, critical Pedagogy, and collaboration: A case forcyborg writing. Computers and the Humanities, 29(6), 431-448.*Wisher, R. A., & Priest, A. N. (1998). Cost-effectiveness of audio teletraining for the U.S.Army National Guard. American Journal of Distance Education, 12(1), 38-51.*Wishner, R. A., Curnow, C. K., & Seidel, R. J. (2001). Knowledge retention as a latent outcomemeasure in distance learning. American Journal of Distance Education, 15(3), 20-35.

Meta-Analysis of the Comparative Literature in Distance Education 45Appendix ADE Codebook with k, g and pThe codebook is divided into the following sections:Section 1 - Identification of Studies (#1-5)Section 2 - Outcome Features (#6-8)Section 3 - Methodological Features (#9-16)Section 4 - Course Design (#17-35)Section 5 - Demographics (#36-46)Section 6 - Pedagogy (#49-51)Section 1 - Identification of Studies1. Study Number: Each article (include or exclude) is given a unique three/four digit number.2. Finding Number: Each finding within a study gets a two digit number starting at 01 for the first finding in each study.3. Author Name: Each study is identified in the order of authorship.4. Year of Publication: Each study is identified by year of publication. In case of a dissertation and a published article. The published articlewill be used.Section 2 - Outcome Features5. Outcome Type: (Describes the basic outcomes explored by the finding). Success rate is the percent of people who completed the course(reverse sign if dropout data is given).1. Achievement k = 251, g = +.0799, p < .05; outliers removed, k = 248, g = +.0551, p < .052. Retention k = 73, g = -.1034, p < .053. Attitude towards course k = 66, g = -.0087, p < .054. Attitude towards the technology k = 24, g = +.1498, p < .055. Attitude towards the subject matter k = 11, g = -.1876, p > .056. Attitude towards the instructor k = 29, g = -.1720, p < .05999. Other k = 56, g = .2341, p < .05

Meta-Analysis of the Comparative Literature in Distance Education 466. Whose Outcome: (Describes the person or persons whose behavior is measured by the outcome instrument).1. Group k = 02. Individual k = 248, g = 0.0551, p < .013. Teacher k = 0999. Missing k = 07. Outcome Measure: (Describes the validity and/or reliability of the outcome measure as a function of its source. Teacher/researcher when theinstructor is also part of the team conducting the research).1. Standardized test k = 2, g = -.1121, p < .052. Researcher-made test k = 24, g = +.2552, p < .053. Teacher-made test k = 205, g = +.0268, p < .054. Teacher/Researcher made test k = 16, g = +.0186, p > .05999. Missing k = 1, g = +.6722, p < .058. Effect Size: (Describes the way in which the statistics were reported by the author. “Calculated” means that we are able to extract M, SD, N.“Not calculated” means that we have to make a best guess estimate using probabilities, etc.)0. Calculated k = 163, g = +.0921, p < .051. Not calculated k = 85, g = -.0807, p < .05Section 3 - Methodological Features9. Type of Publication: (Describes the source of the study. A “report” is a conference paper, governmental report, university report or ED that isnot a dissertation)1. Journal k = 92, g = +.09762. Book chapter k = 03. Report k = 96, g = +.0662, p < .054. Dissertation k = 60, g = -.0071, p > .05

Meta-Analysis of the Comparative Literature in Distance Education 4710. Treatment Duration (Length of time to which any one given student is exposed to the treatment. This is often equivalent to the length of thecourse. One semester is defined as approximately three months or three credits.)1. Less than one semester k = 28, g = +.0274, p > .052. One semester k = 174, g = +.0368, p < .053. More than one semester k = 23, g = +.1322, p < .01999. Missing k = 23, g = +.0567, p < .0511. Treatment Proximity: (This category identifies whether the compared data was collected for both the control and experimental conditionsover the same period of time. An example of when they are not the same would be when data from 1998 (DE) was compared to data from1999(control).1. Same time period k = 208, g = +.0107, p > .052. Different time period k = 21, g = +.2367, p < .01999. Missing k = 19, g = +1320, p < .0112. Number of Control Conditions (This category identifies whether the experimental data was compared to one control condition or more thanone condition. This could be when we have two face to face groups being compared to one DE group.)1. One control, one DE k = 209, g = +.0230, p > .052. One control, more than one DE k = 22, g = +.1769 , p< .013. One DE, more than one control k = 12, g = -.1265, p< .054. More than one DE and more than one control k = 5, g = +.1506, p< .0113. Instructor equivalence: (This category identifies whether the instructor for both the control and experimental conditions was the same.)1. Same instructor k = 194, g = -.0213, p> .052. Different instructor k = 22, g = +.2369, p< .01999. Missing k = 32, g = +.1105, p< .0114. Student equivalence: (This category describes the amount of statistical control used in sampling procedures. This implies that pretestmeasures were taken prior to the treatment, and are related to the posttest measures. GPA could serve as a valid measurement, but only ifmeasured prior, as opposed to during, the treatment. Gender is not a valid measurement. Random = participants are randomly assigned. There isno pretest. Statistical control = when equivalence is established based on a pretest. Missing or non-equivalent = when the author does not provideany evidence that the groups are in fact equivalent. This includes ‘intact’ groups.) (The categories “3. No control” and “4. Clearly nonequivalentgroups” were eliminated on Sept. 30, 2002.)

Meta-Analysis of the Comparative Literature in Distance Education 481. Random k = 11, g = +.1896, p< .012. Statistical control k = 30, g = +.1551, p< .01999. Missing or non-equivalent (this includes students selfselectingk = 207, g = +.0317, p> .05their coursesection)15. Equivalent time on task: (This category describes the equivalence of time spent by students on any related course work, i.e. study time, classtime, homework.) (DE greater than control would be 66% or larger, DE equals control would be 34 to 65%, DE less than control would be 33%or less)1. DE more than control group k = 8, g = +.1511, p< .012. DE reported/ control group not reported k = 2, g = +,4877, p< .013. DE equal to control group k = 42, g = +.0095, p> .054. Control reported/ DE not reported k = 05. DE less than control group k = 5, g = +.1483, p> .05999. Missing (no information on DE OR control reported) k = 191, g = +.0388, p< .0516. Material Equivalence: (This category describes the equivalence of course materials for the two conditions. Materials include course contentand assignments. If the authors indicate that the course was “redesigned” for the DE version, this indicates that they are different curriculummaterials.)1. Same curriculum materials k = 139, g = +.0173, p>.052. Different curriculum materials k = 31, g = +.1721, p< .01999. Missing k = 78, g = +.0434, p< .05Section 4 - Course Design17. Systematic ID (Instructional Design): (This category describes the amount of conventional instructional design practices and principles thatwere used in developing the course and course materials. This requires some evidence that principles of learning and instruction were at the basisof course design.) (DE greater than control would be 66% or larger, DE equals control would be 34 to 65%, DE less than control would be 33%or less)1. DE more than control group k = 2, g = +.1621, p> .052. DE reported/ control group not reported k = 14, g = +.0525, p> .053. DE equal to control group k = 24, g = +.2155, p< .014. Control reported/ DE not reported k = 05. DE less than control group k = 0999. Neither DE nor Control Group used explicitly or missing k = 208, g = +.0273, p< .05

Meta-Analysis of the Comparative Literature in Distance Education 4918. DE condition: Advance Information: (This category describes whether the distance education students received information on things likeequipment needs, techniques for succeeding in DE environment, technical information PRIOR to the commencement of the course).1. Information received prior to commencement of the course k = 11, g = +.6350, p< .012. Information received at the first course k = 9, g = -.0143, p> .053. No information received. k = 0999. Missing k = 228, g = +.0358, p< .0119. Opportunity for F2F (instructor): (Describes whether the DE students had any opportunities during the course for face to face meetings withthe instructor. A discussion facilitator playing a substantial role in guiding and monitoring participants’ discussions could also be considered aninstructor. Coders can assume that opportunities for face to face were not available if there is sufficient evidence to believe so. For instance, iffrom reading the article we know that the two sites are 100 miles apart or in different states, that may be reasonable evidence that opportunity forf2f was not provided.)1. Yes k = 59, g = +.0550, p< .052. No k = 35, g = +.0779, p< .05999. Missing k = 154, g = +.0513, p< .0120. Opportunity for F2F (peer)(Describes whether the DE students had any opportunities during the course for face to face meetings with otherstudents. This could include DE students amongst themselves, or DE students with control students. Coders can assume that opportunities forface to face were not available if there is sufficient evidence to believe so.)1. Yes k = 123, g = +.0922, p< .012. No k = 15, g = +.0881, p> .05999. Missing k = 110, g = +.0224, p> .0521. Institutional Support for Instructor: (Describes the amount of support given to the instructor in terms of things like; release time, reducedcourse load, additional salary, etc.). (DE greater than control would be 66% or larger, DE equals control would be 34 to 65%, DE less thancontrol would be 33% or less)1. DE more than control group k = 19, g = -.0925, p< .052. DE reported/ control group not reported k = 03. DE equal to control group k = 1, g = -.4977, p> .054. Control reported/ DE not reported k = 05. DE less than control group k = 0999. Neither DE nor Control Group used explicitly or missing k = 228, g = +.0684, p< .05

Meta-Analysis of the Comparative Literature in Distance Education 5022. Technical Support for Students: (This category describes the amount of technical support was given to students for using course relatedtechnology. Examples of “technical support” could include; help desk, on-line help desk, written instructions, computer-based tutorials, demos.)(DE greater than control would be 66% or larger, DE equals control would be 34 to 65%, DE less than control would be 33% or less)1. DE more than control group k = 24, g = -.0195, p> .052. DE reported/ control group not reported k = 14, g = +.1137, p< .053. DE equal to control group k = 7, g = +.0352, p> .054. Control reported/ DE not reported k = 05. DE less than control group k = 0999. Neither DE nor Control Group used explicitly or missing k = 203, g = +.0557, p< .0123. Use of 2-way audio conferencing (This category describes the amount of audio conferencing used in delivery of course. Audio-conferencingis defined as two-way audio connection that links individuals or groups together using regular telephone lines and speaker phones. It involves aminimum of three individuals. If synchronous, code as 3.) (DE greater than control would be 66% or larger, DE equals control would be 34 to65%, DE less than control would be 33% or less)1. DE more than control group k = 1, g = +.3823, p> .052. DE reported/ control group not reported k = 33, g = +.1668, p< .013. DE equal to control group k = 73, g = -.0192, p> .054. Control reported/ DE not reported k = 05. DE less than control group k = 0999. Neither DE nor Control Group used explicitly or missing k = 141, g = +.0268, p> .0524. Use of 2-way video conferencing (This category describes the amount of video conferencing used in delivery of course. If synchronous, codeas 3.) (DE greater than control would be 66% or larger, DE equals control would be 34 to 65%, DE less than control would be 33% or less)1. DE more than control group k = 1, g = +.3823, p> .052. DE reported/ control group not reported k = 12, g = +.1433, p< .013. DE equal to control group k = 34, g = -.0243, p> .054. Control reported/ DE not reported k = 05. DE less than control group k = 0999. Neither DE nor Control Group used explicitly or missing k = 201, g = +.0525, p< .0125. Use of CMC or interactive computer classroom (This category describes the amount of CMC used in delivery of course. CMC conferencing(e.g. First Class), groupware, chat rooms. CMC in broad terms refers to technology used to transmit, store, annotate, or present information. Ifsynchronous, code as 3.) (DE greater than control would be 66% or larger, DE equals control would be 34 to 65%, DE less than control would be33% or less)

Meta-Analysis of the Comparative Literature in Distance Education 511. DE more than control group k = 5, g = +.2131, p< .052. DE reported/ control group not reported k = 22, g = +.2052, p< .013. DE equal to control group k = 14, g = +.0350, p> .054. Control reported/ DE not reported k = 05. DE less than control group k = 0999. Neither DE nor Control Group used explicitly or missing k = 207, g = +.0428, p< .0126. Use of e-mail (This category describes the amount of e-mail used in delivery of course). (DE greater than control would be 66% or larger, DEequals control would be 34 to 65%, DE less than control would be 33% or less)1. DE more than control group k = 9, g = -.2937, p< .012. DE reported/ control group not reported k = 18, g = +.0556, p> .053. DE equal to control group k = 7, g = +.1990, p< .014. Control reported/ DE not reported k = 05. DE less than control group k = 0999. Neither DE nor Control Group used explicitly or missing k = 214, g = +.0577, p< .0127. Use of 1-way broadcast TV or videotape or audiotape (This category describes the amount of broadcast TV and/or video used in the deliveryof the course, such as presentation of course material through TV monitors.) (DE greater than control would be 66% or larger, DE equals controlwould be 34 to 65%, DE less than control would be 33% or less)1. DE more than control group k = 6, g = +.4691, p< .012. DE reported/ control group not reported k = 97, g = +.0632, p< .013. DE equal to control group k = 5, g = -.0565, p> .054. Control reported/ DE not reported k = 05. DE less than control group k = 0999. Neither DE nor Control Group used explicitly or missing k = 140, g = +.0442, p< .0128. Use of web-based course materials(This category describes the amount of web-based course materials used in delivery of course. This refersto instances where students had access to the course materials through a course website or class materials were posted to the web). (DE greaterthan control would be 66% or larger, DE equals control would be 34 to 65%, DE less than control would be 33% or less)1. DE more than control group k = 11, g = +.0902, p> .052. DE reported/ control group not reported k = 21, g = +.1318, p< .013. DE equal to control group k = 3, g = -.0408, p> .054. Control reported/ DE not reported k = 0

Meta-Analysis of the Comparative Literature in Distance Education 525. DE less than control group k = 0999. Neither DE nor Control Group used explicitly or missing k = 213, g = +.0487, p< .0129. Use of telephone (This category describes the amount of telephone communication used in delivery of course. Telephone is defined as twowayaudio connection that links two individuals together using regular telephone equipment. It involves a maximum of two individuals.). (DEgreater than control would be 66% or larger, DE equals control would be 34 to 65%, DE less than control would be 33% or less)1. DE more than control group k = 8, g = +.3072, p< .012. DE reported/ control group not reported k = 16, g = +.0444, p> .053. DE equal to control group k = 17, g = +.2618, p< .014. Control reported/ DE not reported k = 05. DE less than control group k = 0999. Neither DE nor Control Group used explicitly or missing k = 207, g = +.0251, p> .0530. Use of computer-based tutorials/simulations (This category describes the amount of computer based tutorials or simulations used in deliveryof course). (DE greater than control would be 66% or larger, DE equals control would be 34 to 65%, DE less than control would be 33% or less)1. DE more than control group k = 1, g = +.0561, p> .052. DE reported/ control group not reported k = 9, g = +.0912, p> .053. DE equal to control group k = 3, g = -.1436, p> .054. Control reported/ DE not reported k = 05. DE less than control group k = 0999. Neither DE nor Control Group used explicitly or missing k = 235, g = +.0558, p< .0131. Simultaneous Delivery: (This category describes whether the DE condition and control condition received instruction simultaneously)1. Simultaneous delivery k = 79, g = +.0084, p> .052. Not simultaneous k = 112, g = +.0660, p< .01999. Missing k = 57, g = +.0681, p< .0132. Provision for Synchronous technically-mediated Communication /teacher (This category describes the amount of opportunity given forstudents and instructors to communicate synchronously – De condition. This includes: telephone, video-conferencing, chats, etc. and excludesface-to-face.)1. Opportunity for synchronous communication k = 137, g = +.0968, p< .012. No opportunity for synchronous communication k = 44, g = +.0058, p> .05999. Missing k = 67, g = -.0125, p> .05

Meta-Analysis of the Comparative Literature in Distance Education 5333. Provision for Synchronous technically-mediated Communication/students (This category describes the amount of opportunity given forstudents in the DE condition to communicate synchronously. This could include DE students amongst themselves, or DE students with controlstudents. This includes: telephone, video-conferencing, chats, etc. and excludes face-to-face.)1. Opportunity for synchronous communication k = 118, g = +.0465, p< .012. No opportunity for synchronous communication k = 52, g = +.0728, p< .01999. Missing k = 78, g = +.0555, p< .0534. Cost of Course Delivery: (This category describes the degree to which the course delivery methods required equal or unequal amounts offinancial expenditure with regards to start up costs.)1. DE more expensive than control k = 14, g = -.0072, p> .052. DE reported / control not reported k = 8, g = +.2249, p< .053. DE equally expensive as control k = 11, g = -.1714, p< .054. Control reported/ DE not reported k = 05. DE less expensive than control k = 8, g = +.2297, p< .01999. Missing k = 207, g = +.0535, p< .0135. Purpose for Offering DE: (This category describes the reason given by the course developers as to why the distance delivery option was madeavailable. If the course is offered solely for the purpose of a research study = 999. If more than one category is applicable, select the mostimportant one if possible. Otherwise, code as 6.)1. Flexibility of schedule or travel k = 48, g = +.0290, p> .052. Preferred media approach k = 1, g = -.9023, p< .053. Access to expertise (teacher/program) k = 48, g = -.0096, p> .054. Special needs students k = 1, g = +.2669, p> .055. Efficient delivery or cost savings k = 31, g = +.0446, p> .056. Multiple reasons. Specify: ____________ k = 14, g = +.3373, p< .01999. Missing k = 105, g = +.0367, p< .05Section 5 – Demographics36. Instructor Experience with DE: (This category describes the amount of prior experience teaching via distance that the instructor of the DEcondition had. This should be explicitly reported. However, if the study was done over more than one semester, and reports that the sameinstructor taught more than one group, we consider that the instructor has experience at the second group.)1. Yes k = 16, g = +.0358, p> .05

Meta-Analysis of the Comparative Literature in Distance Education 542. No k = 11, g = -.0982, p> .05999. Missing k = 221, g = +.0607, p< .0137. Instructor Experience with technologies used (This category describes the amount of prior experience the instructor of the DE condition hadin using the technologies that were part of the present course. This should be explicitly reported. However, if the study was done over more thanone semester, and reports that the same instructor taught more than one group, we consider that the instructor has experience at the secondgroup.)1. Yes k = 23, g = +.1055, p < .012. No k = 5, g = -.0834, p > .05999. Missing k = 220, g = +.0530, p< .0138. Students’ Experience with DE: (This category describes the amount of prior experience with distance learning that the students of the DEcondition had).1. Yes k = 2, g = +.6324, p< .012. No k = 6, g = +.2820, p< .01999. Missing k = 240, g = +.0453, p< .0139. Students’ Experience with technologies used (This category describes the amount of prior experience the students of the DE condition had inusing the technologies that were part of the present course).1. Yes k = 5, g = +.3092, p< .012. No k = 3, g = +.6738, p< .01999. Missing k = 240, g = +.0442, p< .0140. Types of Control Learners: (This category describes the control students level of education. Professionals= training that is provided as part ofa groups professional development. If more than one category is applicable, select the one that represents the highest proportion of learners in thestudy.)1. Grade School (K-12) k = 22, g = +.1210, p< .012. Undergraduate k = 155, g = +.0323, p> .053. Graduate k = 38, g = +.0848, p< .014. Military k = 10, g = +.2050, p< .015. Industry/business k = 4, g = +.2690, p< .016. Professionals (e.g., doctors) k = 10, g = -.0525, p> .05999. Missing or more than one category. Specify : ________________ k = 9, g = +.0138, p> .05

Meta-Analysis of the Comparative Literature in Distance Education 5541. Types of DE Learners: (This category describes the DE students level of education. Professionals= training that is provided as part of agroups professional development. If more than one category is applicable, select the one that represents the highest proportion of learners in thestudy.)1. Grade School (K-12) k = 22, g = +.1210, p< .012. Undergraduate k = 154, g = +.0311, p> .053. Graduate k = 29, g = +.0886, p< .014. Military k = 10, g = +.2050, p< .015. Industry/business k = 14, g = +.2027, p< .056. Professionals (e.g., doctors) k = 10, g = -.0525, p> .05999. Missing or more than one category. Specify : ________________ k = 19, g = +.0138, p> .0542. Learner ability: (This category describes the ability level of students in both conditions. Higher ability could be indicated by students beinglabeled “gifted”, lower ability could be indicated by referring to “remediation” education. “ Equal” means that both groups were made up ofmixed ability learners. Equal learner ability should NOT be assumed and needs to be explicitly stated. (DE greater than control would be 66% orlarger, DE equals control would be 34 to 65%, DE less than control would be 33% or less)1. DE students higher ability than control group k = 6, g = +.0648, p> .052. DE reported/ control group not reported k = 2, g = +.4177, p< .013. DE students’ ability equal to control group k = 39, g = +.0651, p< .054. Control reported/ DE not reported k = 05. DE students lower ability than control group k = 0999. Missing (no information on ability level of DE OR control reported) k = 201, g = +.0319, p< .0543. Setting: (This category describes the geographical location of the students in both conditions. The coder should make no assumptions. If citiesare reported but the coder is unaware of whether they are urban or rural, the city names will be indicated by the coder on the spreadsheet.)1. DE urban and control rural k = 1, g = +.5923, p< .012. DE urban and control urban k = 22, g = +.1551, p< .013. DE reported/ Control not reported k = 6, g = +.1157, p> .054. DE rural and control urban k = 3, g = -.0250, p> .055. DE rural and control rural k = 46, g = -.0569, p< .056. Control reported. / DE not reported k = 6, g = +.3281, p< .01999. Missing (no information on setting of DE or control groups reported) k = 164, g = +.0522, p< .01

Meta-Analysis of the Comparative Literature in Distance Education 5644. Subject Matter: (This category describes the content domain of the course materials. This refers to main subject of the curriculum beingtaught. Computer applications = learning specific software)1. Math (including stats and algebra) k = 25, g = –.0479, p > .052. Languages arts (collapsed) k = 18, g = +.1539, p < .013. Science and Engineering (collapsed) k = 60, g = +.1331, p < .014. Humanities and Education (collapsed) k = 44, g = –.0046, p > .055. Geography (included in 4)6. Computer science (information technology) (included in 3)7. Computer applications (included in 3)8. Education (included in 4)9. Medicine or nursing (histology) (included in 3)10. Military training k = 11, g = +.0529, p > .0511. Business k = 41, g = –.1113, p < .0112. Engineering (included in 3)13. Other (specify) (included in 999)999. Missing k = 49, g = +.0921, p .0545. Attrition Rates: (This category describes the amount of students who dropped the course before it was completed) (DE greater than controlwould be 66% or larger, DE equals control would be 34 to 65%, DE less than control would be 33% or less)1. DE more than control group k = 6, g = -.1468, p> .052. DE reported/ control group not reported k = 2, g = +.2097, p> .053. DE equal to control group k = 55, g = +.1350, p< .014. Control reported/ DE not reported k = 05. DE less than control group k = 1, g = +.4211, p> .05999. Missing (no information on DE OR control reported) k = 184, g = +.0461, p< .0146. Average Class Size (This category describes the size of the student groups. E.g. If one DE group of 500 students is compared to 10 controlgroups of 50 students, use average of 50 students for control i.e. DE would be larger than control in this case. Class size is not in relation to theeffect size, but in relation to the study design. That is, if k’s were collapsed to calculate ES, the coder should look at original number of studentsper group as reported in the study rather than k’s used for ES. (DE greater than control would be 66% or larger, DE equals control would be 34 to65%, DE less than control would be 33% or less)1. DE larger than control k = 35, g = +.0052, p> .05

Meta-Analysis of the Comparative Literature in Distance Education 572. DE equal to control k = 84, g = -.0044, p> .053. DE smaller than control k = 88, g = +.0190, p> .054. Miscodes k = 5, g = +.1506, p< .01999. Missing k = 36, g = +.1180, p< .0147. Average age: (This category describes the age equivalence of the two conditions. For K-12 students, the assumption about age equalitybetween experimental and control students is justified and allowed) (DE greater than control would be 66% or larger, DE equals control would be34 to 65%, DE less than control would be 33% or less)1. DE more than control group k = 23, g = +.1921, p< .012. DE reported/ control group not reported k = 5, g = +.1506, p< .013. DE equal to control group k = 49, g = +.1075, p< .014. Control reported/ DE not reported k = 05. DE less than control group k = 1, g = -.5075, p> .05999. Missing (no information on DE OR control reported) k = 169, g = +.0075, p> .0548. Gender: (This category compares the gender equivalence of the two conditions. Results will be reported in terms of the Male/Female ratio.)(DE greater than control would be 66% or larger, DE equals control would be 34 to 65%, DE less than control would be 33% or less)1. DE more than control group k = 02. DE reported/ control group not reported k = 6, g = +.1537, p< .013. DE equal to control group k = 43, g = +.2116, p< .014. Control reported/ DE not reported k = 05. DE less than control group k = 3, g = +.1393, p> .05999. Missing (no information on DE OR control reported) k = 196, g = +.0016, p> .05Section 6 – Pedagogy49. Teacher/Student Contact Encouraged: (This category describes the degree to which student/teacher contact was encouraged through courseactivities or by course design. Indicators of “contact encouraged” include things like; regularly scheduled office hours, class discussions, one-ononetutoring by instructor, survey data where students rate degree of contact with teacher, etc.) (DE greater than control would be 66% or larger,DE equals control would be 34 to 65%, DE less than control would be 33% or less)1. DE more than control group k = 11, g = +.3079, p< .012. DE reported/ control group not reported k = 11, g = +.3184, p< .013. DE equal to control group k = 37, g = +.0223, p> .054. Control reported/ DE not reported k = 0

Meta-Analysis of the Comparative Literature in Distance Education 585. DE less than control group k = 15, g = +.3118, p < .01999. Neither DE nor Control Group used explicitly or missing k = 174, g = +.0165, p> .0550. Student/Student Contact Encouraged: (This category describes the degree to which student/student contact was encouraged through courseactivities or by course design. Indicators of “contact encouraged” include things like; group projects, group discussions, group chats, peertutoring,CMC, etc.) (DE greater than control would be 66% or larger, DE equals control would be 34 to 65%, DE less than control would be33% or less)1. DE more than control group k = 14, g = +.1067, p> .052. DE reported/ control group not reported k = 10, g = +.3230, p< .013. DE equal to control group k = 27, g = +.0339, p> .054. Control reported/ DE not reported k = 05. DE less than control group k = 7, g = +.6975, p< .01999. Neither DE nor Control Group used explicitly or missing k = 190, g = +.0161, p> .0551. Problem-based Learning: (This category describes the degree to which course activities or course design promoted active forms of learnerengagement, or in other words “learning by doing”. Indicators include things like: students working together to solve real world problems, activelearning, discovery learning, authentic performance-based assessments, etc. Indicators that problem-based learning was NOT used; hypotheticalproblems/case studies, lecture-based course, reading and completing assignments individually, etc.) (DE greater than control would be 66% orlarger, DE equals control would be 34 to 65%, DE less than control would be 33% or less)1. DE more than control group k = 6, g = +.0257, p> .052. DE reported/ control group not reported k = 6, g = +.2773, p< .013. DE equal to control group k = 12, g = -.0367, p> .054. Control reported/ DE not reported k = 05. DE less than control group k = 0999. Neither DE nor Control Group used explicitly or missing k = 224, g = +.0440, p< .01