Eugene Borokhovski

A Meta-Analysis of Three Types of Interaction Treatments in Distance Education:

This meta-analysis of the experimental literature of distance education (DE) compares different types of interaction treatments (ITs) with other DE instructional treatments. ITs are the instructional and/or media conditions designed into DE courses, which are intended to facilitate student–student (SS), student–teacher (ST), or student–content (SC) interactions. Seventy-four DE versus DE studies that contained at least one IT are included in the meta-analysis, which yield 74 achievement effects. The effect size valences are structured so that the IT or the stronger IT (i.e., in the case of two ITs) serve as the experimental condition and the other treatment, the control condition. Effects are categorized as SS, ST, or SC. After adjustment for methodological quality, the overall weighted average effect size for achievement is 0.38 and is heterogeneous. Overall, the results support the importance of the three types of ITs and strength of ITs is found to be associated with increasing achievement outcomes. A strong association is found between strength and achievement for asynchronous DE courses compared to courses containing mediated synchronous or face-to-face interaction. The results are interpreted in terms of increased cognitive engagement that is presumed to be promoted by strengthening ITs in DE courses.

What Forty Years of Research Says About the Impact of Technology on Learning: A Second-Order Meta-Analysis and Validation Study

This research study employs a second-order meta-analysis procedure to summarize 40 years of research activity addressing the question, does computer technology use affect student achievement in formal face-to-face classrooms as compared to classrooms that do not use technology? A study-level meta-analytic validation was also conducted for purposes of comparison. An extensive literature search and a systematic review process resulted in the inclusion of 25 meta-analyses with minimal overlap in primary literature, encompassing 1,055 primary studies. The random effects mean effect size of 0.35 was significantly different from zero. The distribution was heterogeneous under the fixed effects model. To validate the second-order meta-analysis, 574 individual independent effect sizes were extracted from 13 out of the 25 meta-analyses. The mean effect size was 0.33 under the random effects model, and the distribution was heterogeneous. Insights about the state of the field, implications for technology use, and prospects for future research are discussed.

Interaction in distance education and online learning: using evidence and theory to improve practice

In a recent meta-analysis of distance and online learning, Bernard et al. (2009) quantitatively verified the importance of three types of interaction: among students, between the instructor and students, and between students and course content. In this paper we explore these findings further, discuss methodological issues in research and suggest how these results may foster instructional improvement. We highlight several evidence-based approaches that may be useful in the next generation of distance and online learning. These include principles and applications stemming from the theories of self-regulation and multimedia learning, research-based motivational principles and collaborative learning principles. We also discuss the pedagogical challenges inherent in distance and online learning that need to be considered in instructional design and software development.

A Review of e-Learning in Canada: A Rough Sketch of the Evidence, Gaps and Promising Directions

This review provides a rough sketch of the evidence, gaps and promising directions in e-learning from 2000 onwards, with a particular focus on Canada. We searched a wide range of sources and document types to ensure that we represented, comprehensively, the arguments surrounding e-learning. Overall, there were 2,042 entries in our database, of which we reviewed 1,146, including all the Canadian primary research and all scholarly reviews of the literature. In total, there were 726 documents included in our review: 235 – general public opinion; 131 – trade/practitioners’ opinion; 88 – policy documents; 120 – reviews; and 152 – primary empirical research. The Argument Catalogue codebook included the following eleven classes of variables: 1) Document Source; 2) Areas/Themes of e-learning; 3) Value/Impact; 4) Type of evidence; 5) Research design; 6) Area of applicability; 7) Pedagogical implementation factors; 8) A-priori attitudes; 9) Types of learners; 10) Context; and 11) Technology Factors. We examined the data from a number of perspectives, including their quality as evidence. In the primary research literature, we examined the kinds of research designs that were used. We found that over half of the studies conducted in Canada are qualitative in nature, while the rest are split in half between surveys and quantitative studies (correlational and experimental). When we looked at the nature of the research designs, we found that 51% are qualitative case studies and 15.8% are experimental or quasi-experimental studies. It seems that studies that can help us understand “what works” in e-learning settings are underrepresented in the Canadian research literature. The documents were coded to provide data on outcomes of e-learning (we also refer to them as “impacts” of e-learning). Outcomes/impacts are the perceived or measured benefits of e-learning, whereas predictors are the conditions or features of e-learning that can potentially affect the outcomes/impacts. The impacts were coded on a positive to negative scale and included: 1) achievement; 2) motivation/satisfaction; 3) interactivity/ communication; 4) meeting social demands; 5) retention/attrition; 6) learning flexibility; and 7) cost. Based on an analysis of the correlations among these impacts, we subsequently collapsed them (all but cost) into a single impact scale ranging from –1 to +1. We found, generally, that the perception of impact or actual measured impact varies across the types of documents. They appear to be lower in general opinion documents, practitioner documents and policy making reports than in scholarly reviews and primary research. While this represents an expression of hope for positive impact, on the one hand, it possibly represents reality, on the other. Where there were sufficient documents to examine and code, impact was high across each of the CCL Theme Areas. Health and Learning was the highest, with a mean of 0.80 and Elementary/Secondary was the lowest, with a mean of 0.77. However, there was no significant difference between these means. The impact of e-learning and technology use was highest in distance education, where its presence is required (Mean = 0.80) and lowest in face-to-face instructional settings (Mean = 0.60) where its presence is not required. Network-based technologies (e.g., Internet, Web-based, CMC) produced a higher impact score (Mean = 0.72) than straight technology integration in educational settings (Mean = 0.66), although this difference was considered negligible. Interestingly, among the Pedagogical Uses of Technology, student applications (i.e., students using technology) and communication applications (both Mean = 0.78) had a higher impact score than instructional or informative uses (Mean = 0.63). This result suggests that the student manipulation of technology in achieving educational goals is preferable to teacher manipulation of technology. In terms of predictor variables (professional training, course design, infrastructure/ logistics, type of learners [general population, special needs, gifted], gender issues and ethnicity/race/religion/aboriginal, location, school setting, context of technology use, type of tool used and pedagogical function of technology) we found the following: professional development was underrepresented compared to issues of course design and infrastructure/ logistics; most attention is devoted to general population students, with little representation of special needs, the gifted students, issues of gender or ethnic/race/religious/aboriginal status; the greatest attention is paid to technology use in distance education and the least attention paid to the newly emerging area of hybrid/blended learning; the most attention is paid to networked technologies such as the Internet, the WWW and CMC and the least paid to virtual reality and simulations. Using technology for instruction and using technology for communication are the two highest categories of pedagogical use. In the final stage, the primary e-learning studies from the Canadian context that could be summarized quantitatively were identified. We examined 152 studies and found a total of 7 that were truly experimental (i.e., random assignment with treatment and control groups) and 10 that were quasi-experimental (i.e., not randomized but possessing a pretest and a posttest). For these studies we extracted 29 effect sizes or standardized mean differences, which were included in the composite measure. The mean effect size was +0.117, a small positive effect. Approximately 54% of the e-learning participants performed at or above the mean of the control participants (50 th percentile), an advantage of 4%. However, the heterogeneity analysis was significant, indicating that the effect sizes were widely dispersed. It is clearly not the case that e-learning is always the superior condition for educational impact. Overall, we know that research in e-learning has not been a Canadian priority; the culture of educational technology research, as distinct from development, has not taken on great import. In addition, there appears to have been a disproportionate emphasis on qualitative research in the Canadian e-learning research culture. We noted that there are gaps in areas of research related to early childhood education and adult education. Finally, we believe that more emphasis must be placed on implementing longitudinal research, whether qualitative or quantitative (preferably a mixture of the two), and that all development efforts be accompanied by strong evaluation components that focus on learning impact. It is a shame to attempt innovation and not be able to tell why it works or doesn’t work. In this sense, the finest laboratories for e-learning research are the institutions in which it is being applied. Implications for K-12 Practitioners When implemented appropriately, technology tools are beneficial to students’ learning, and may facilitate the development of higher order thinking skills. Student manipulation of technology in achieving the goals of education is preferable to teacher manipulation of technology. Teachers need to be aware of differences between instructional design for e-learning as compared to traditional face-to-face situations. Immediate, extensive, and sustained support should be offered to teachers in order to make the best out of e-learning. Implications for Post-Secondary Some educators suggest that e-learning has the potential to transform learning, but there is limited empirical research to assess the benefits. Post-secondary education would benefit from a Pan-Canadian plan to assess the impact of e-learning initiatives. It is important that instructional design match the goals and potential of e-learning. Research is needed to determine the feasibility and effectiveness of such things as learning objects and multimedia applications. Properly implemented computer mediated communication can enrich the learning environment; help reduce low motivation and feelings of isolation in distance learners. E-learning appears to be more effective in distance education, where technology use is required than in face-to-face instructional settings. Implications for Policy Makers Effective and efficient implementation of e-learning technologies represents new, and difficult, challenges to practitioners, researchers, and policymakers. The term e-learning has been used to describe many different applications of technology, which may be implemented in a wide variety of ways (some of which are much more beneficial than others). School administrators must balance the needs of all stakeholders, and the cost-benefit ratios of technology tools, in deciding not only which technologies to use, but also when and how to implement new technologies. Traditional methods of instructional design and school administration must be adjusted to deal with the demands of distance education and other contexts of technology use. Professional education, development, and training for educators must ensure that teachers will be equipped to make optimal pedagogical use of new methods.

Strategies for Teaching Students to Think Critically A Meta-Analysis

Critical thinking (CT) is purposeful, self-regulatory judgment that results in interpretation, analysis, evaluation, and inference, as well as explanations of the considerations on which that judgment is based. This article summarizes the available empirical evidence on the impact of instruction on the development and enhancement of critical thinking skills and dispositions and student achievement. The review includes 341 effects sizes drawn from quasi- or true-experimental studies that used standardized measures of CT as outcome variables. The weighted random effects mean effect size (g+) was 0.30 (p < .001). The collection was heterogeneous (p < .001). Results demonstrate that there are effective strategies for teaching CT skills, both generic and content specific, and CT dispositions, at all educational levels and across all disciplinary areas. Notably, the opportunity for dialogue, the exposure of students to authentic or situated problems and examples, and mentoring had positive effects on CT skills.

The effects of technology use in postsecondary education

This meta-analysis is a study of the experimental literature of technology use in postsecondary education from 1990 up to 2010 exclusive of studies of online or distance education previously reviewed by Bernard et al. (2004). It reports the overall weighted average effects of technology use on achievement and attitude outcomes and explores moderator variables in an attempt to explain how technology treatments lead to positive or negative effects. Out of an initial pool of 11,957 study abstracts, 1105 were chosen for analysis, yielding 879 achievement and 181 attitude effect sizes after pre-experimental designs and studies with obvious methodological confounds were removed. The random effects weighted average effect size for achievement was g+ = 0.27, k = 879, p < .05, and for attitude outcomes it was g+ = 0.20, k = 181, p < .05. The collection of achievement outcomes was divided into two sub-collections, according to the amount of technology integration in the control condition. These were no technology in the control condition (k = 479) and some technology in the control condition (k = 400). Random effects multiple meta-regression analysis was run on each sub-collection revealing three significant predictors (subject matter, degree of difference in technology use between the treatment and the control and pedagogical uses of technology). The set of predictors for each sub-collection was both significant and homogeneous. Differences were found among the levels of all three moderators, but particularly in favor of cognitive support applications. There were no significant predictors for attitude outcomes.

Are contextual and designed student–student interaction treatments equally effective in distance education?

This systematic review draws from and builds upon the results of a meta-analysis of the achievement effects of three types of interaction treatments in distance education: student–student, student–teacher, and student–content (Bernard et al., Review of Educational Research, 79(3), 1243–1289, 2009). This follow-up study considers two forms of student–student interaction treatments, contextual interaction and designed interaction. Typical contextual interaction treatments contain the necessary conditions for student–student interaction to occur, but are not intentionally designed to create collaborative learning environments. By contrast, designed interaction treatments are intentionally implemented collaborative instructional conditions for increasing student learning. Our meta-analysis compared the effect of these two types of interaction treatments on student achievement outcomes. The results favored designed interaction treatments over contextual interaction treatments. Examples of designed interaction treatments and a discussion of study results and their potential implications for research and instruction in distance education and online learning are presented.

Technology-supported student interaction in post-secondary education

The present study extends the results of a larger meta-analysis that addressed the effects of technology use on student achievement and attitudes in postsecondary education. The focus of the current meta-analysis is the use of technology to enable instructional conditions that promote collaborative interactions among learners. More specifically, it aims to compare the impact of designed interaction treatments (i.e., collaborative activities intentionally built into course design) and contextual interaction treatments (i.e., course conditions that result in high levels of student-student interaction but are not intentionally designed to promote collaboration) on student learning outcomes. Results indicate that designed treatments outperform contextual treatments ( g ? ?=?0.52, k?=?25 vs. g ? ?=?0.11, k?=?20, QBetween?=?7.91, p?<?.02) on measures of achievement, emphasizing the importance of planning and instructional design in technology integration in postsecondary education. The results are discussed in relation to the literature of student-student interaction and collaborative learning. Meta-analysis of designed and contextual interaction treatments.Covers studies published from 1990 up to 2010.Designed (k?=?25) versus contextual interaction treatments (k?=?20).Comparison was significant ( g ? ?=?0.52, vs. g ? ?=?0.11, QBetween?=?7.91, p?<?.02).Moderator variable analysis yielded positive outcomes for cognitive support tools.

An Extended Systematic Review of Canadian Policy Documents on e-Learning: What We’re Doing and Not Doing

This systematic review builds upon the work of Authors (2006) and McGreal and Anderson (2007). It seeks to provide a synthesis and discussion of publicly available government policy documents with regard to e-learning in Canada. There is general consensus, both in public opinion and in the research literature, that the educational practices associated with rapidly advancing computer information technologies are gaining popularity and are expected to be increasingly effective in enhancing learning. The purpose of this review is to uncover and describe areas of commonality and inconsistency in e-learning policy documents dated from 2000 to 2010, and to determine where discussions about e-learning are lacking. In total, 138 policy documents from Canadian provinces and territories and several federal agencies were retrieved and analyzed using prescriptive and emergent coding approaches. The review confirmed that Canadian policy makers view technology as offering potential benefits to learners, but also revealed a troubling lack of specific details, consistency and coordination in facilitating the development of e-learning to fulfill these optimistic expectations.

Detecting bias in meta-analyses of distance education research: big pictures we can rely on

This article has two interrelated purposes. The first is to explain how various forms of bias, if introduced during any stage of a meta-analysis, can provide the consumer with a misimpression of the state of a research literature. Five of the most important bias-producing aspects of a meta-analysis are presented and discussed. Second, armed with this information, we examine 15 meta-analyses of the literatures of distance education (DE), online learning (OL), and blended learning (BL), conducted from 2000 to 2014, with the intention of assessing potential sources of bias in each. All of these meta-analyses address the question: “How do students taking courses through DE, OL, and BL compare to students engaged in pure classroom instruction in terms of learning achievement outcomes?” We argue that questions asked by primary researchers must change to reflect issues that will drive improvements in designing and implementing DE, OL, and BL courses.