Richard F. Schmid

What Forty Years of Research Says About the Impact of Technology on Learning: A Second-Order Meta-Analysis and Validation Study

This research study employs a second-order meta-analysis procedure to summarize 40 years of research activity addressing the question, does computer technology use affect student achievement in formal face-to-face classrooms as compared to classrooms that do not use technology? A study-level meta-analytic validation was also conducted for purposes of comparison. An extensive literature search and a systematic review process resulted in the inclusion of 25 meta-analyses with minimal overlap in primary literature, encompassing 1,055 primary studies. The random effects mean effect size of 0.35 was significantly different from zero. The distribution was heterogeneous under the fixed effects model. To validate the second-order meta-analysis, 574 individual independent effect sizes were extracted from 13 out of the 25 meta-analyses. The mean effect size was 0.33 under the random effects model, and the distribution was heterogeneous. Insights about the state of the field, implications for technology use, and prospects for future research are discussed.

Concept Mapping as an Instructional Strategy for High School Biology

AbstractIn this study, concept mapping was assessed as an instructional strategy for use by high school students in learning biology concepts. Concept mapping, which requires learners to plot concepts and their interrelations in a meaningful organizational network, was compared with an established curriculum approach and tested in an interaction with learners of varying verbal ability. The design emphasized external validity. As hypothesized, concept mapping facilitated low-ability learners’ performance, but only on higher level, relational knowledge. The usefulness of the technique was discussed in terms of its ability to individualize and raise the quality of learning with little extra effort or resource costs to the instructional system.

A Review of e-Learning in Canada: A Rough Sketch of the Evidence, Gaps and Promising Directions

This review provides a rough sketch of the evidence, gaps and promising directions in e-learning from 2000 onwards, with a particular focus on Canada. We searched a wide range of sources and document types to ensure that we represented, comprehensively, the arguments surrounding e-learning. Overall, there were 2,042 entries in our database, of which we reviewed 1,146, including all the Canadian primary research and all scholarly reviews of the literature. In total, there were 726 documents included in our review: 235 – general public opinion; 131 – trade/practitioners’ opinion; 88 – policy documents; 120 – reviews; and 152 – primary empirical research. The Argument Catalogue codebook included the following eleven classes of variables: 1) Document Source; 2) Areas/Themes of e-learning; 3) Value/Impact; 4) Type of evidence; 5) Research design; 6) Area of applicability; 7) Pedagogical implementation factors; 8) A-priori attitudes; 9) Types of learners; 10) Context; and 11) Technology Factors. We examined the data from a number of perspectives, including their quality as evidence. In the primary research literature, we examined the kinds of research designs that were used. We found that over half of the studies conducted in Canada are qualitative in nature, while the rest are split in half between surveys and quantitative studies (correlational and experimental). When we looked at the nature of the research designs, we found that 51% are qualitative case studies and 15.8% are experimental or quasi-experimental studies. It seems that studies that can help us understand “what works” in e-learning settings are underrepresented in the Canadian research literature. The documents were coded to provide data on outcomes of e-learning (we also refer to them as “impacts” of e-learning). Outcomes/impacts are the perceived or measured benefits of e-learning, whereas predictors are the conditions or features of e-learning that can potentially affect the outcomes/impacts. The impacts were coded on a positive to negative scale and included: 1) achievement; 2) motivation/satisfaction; 3) interactivity/ communication; 4) meeting social demands; 5) retention/attrition; 6) learning flexibility; and 7) cost. Based on an analysis of the correlations among these impacts, we subsequently collapsed them (all but cost) into a single impact scale ranging from –1 to +1. We found, generally, that the perception of impact or actual measured impact varies across the types of documents. They appear to be lower in general opinion documents, practitioner documents and policy making reports than in scholarly reviews and primary research. While this represents an expression of hope for positive impact, on the one hand, it possibly represents reality, on the other. Where there were sufficient documents to examine and code, impact was high across each of the CCL Theme Areas. Health and Learning was the highest, with a mean of 0.80 and Elementary/Secondary was the lowest, with a mean of 0.77. However, there was no significant difference between these means. The impact of e-learning and technology use was highest in distance education, where its presence is required (Mean = 0.80) and lowest in face-to-face instructional settings (Mean = 0.60) where its presence is not required. Network-based technologies (e.g., Internet, Web-based, CMC) produced a higher impact score (Mean = 0.72) than straight technology integration in educational settings (Mean = 0.66), although this difference was considered negligible. Interestingly, among the Pedagogical Uses of Technology, student applications (i.e., students using technology) and communication applications (both Mean = 0.78) had a higher impact score than instructional or informative uses (Mean = 0.63). This result suggests that the student manipulation of technology in achieving educational goals is preferable to teacher manipulation of technology. In terms of predictor variables (professional training, course design, infrastructure/ logistics, type of learners [general population, special needs, gifted], gender issues and ethnicity/race/religion/aboriginal, location, school setting, context of technology use, type of tool used and pedagogical function of technology) we found the following: professional development was underrepresented compared to issues of course design and infrastructure/ logistics; most attention is devoted to general population students, with little representation of special needs, the gifted students, issues of gender or ethnic/race/religious/aboriginal status; the greatest attention is paid to technology use in distance education and the least attention paid to the newly emerging area of hybrid/blended learning; the most attention is paid to networked technologies such as the Internet, the WWW and CMC and the least paid to virtual reality and simulations. Using technology for instruction and using technology for communication are the two highest categories of pedagogical use. In the final stage, the primary e-learning studies from the Canadian context that could be summarized quantitatively were identified. We examined 152 studies and found a total of 7 that were truly experimental (i.e., random assignment with treatment and control groups) and 10 that were quasi-experimental (i.e., not randomized but possessing a pretest and a posttest). For these studies we extracted 29 effect sizes or standardized mean differences, which were included in the composite measure. The mean effect size was +0.117, a small positive effect. Approximately 54% of the e-learning participants performed at or above the mean of the control participants (50 th percentile), an advantage of 4%. However, the heterogeneity analysis was significant, indicating that the effect sizes were widely dispersed. It is clearly not the case that e-learning is always the superior condition for educational impact. Overall, we know that research in e-learning has not been a Canadian priority; the culture of educational technology research, as distinct from development, has not taken on great import. In addition, there appears to have been a disproportionate emphasis on qualitative research in the Canadian e-learning research culture. We noted that there are gaps in areas of research related to early childhood education and adult education. Finally, we believe that more emphasis must be placed on implementing longitudinal research, whether qualitative or quantitative (preferably a mixture of the two), and that all development efforts be accompanied by strong evaluation components that focus on learning impact. It is a shame to attempt innovation and not be able to tell why it works or doesn’t work. In this sense, the finest laboratories for e-learning research are the institutions in which it is being applied. Implications for K-12 Practitioners When implemented appropriately, technology tools are beneficial to students’ learning, and may facilitate the development of higher order thinking skills. Student manipulation of technology in achieving the goals of education is preferable to teacher manipulation of technology. Teachers need to be aware of differences between instructional design for e-learning as compared to traditional face-to-face situations. Immediate, extensive, and sustained support should be offered to teachers in order to make the best out of e-learning. Implications for Post-Secondary Some educators suggest that e-learning has the potential to transform learning, but there is limited empirical research to assess the benefits. Post-secondary education would benefit from a Pan-Canadian plan to assess the impact of e-learning initiatives. It is important that instructional design match the goals and potential of e-learning. Research is needed to determine the feasibility and effectiveness of such things as learning objects and multimedia applications. Properly implemented computer mediated communication can enrich the learning environment; help reduce low motivation and feelings of isolation in distance learners. E-learning appears to be more effective in distance education, where technology use is required than in face-to-face instructional settings. Implications for Policy Makers Effective and efficient implementation of e-learning technologies represents new, and difficult, challenges to practitioners, researchers, and policymakers. The term e-learning has been used to describe many different applications of technology, which may be implemented in a wide variety of ways (some of which are much more beneficial than others). School administrators must balance the needs of all stakeholders, and the cost-benefit ratios of technology tools, in deciding not only which technologies to use, but also when and how to implement new technologies. Traditional methods of instructional design and school administration must be adjusted to deal with the demands of distance education and other contexts of technology use. Professional education, development, and training for educators must ensure that teachers will be equipped to make optimal pedagogical use of new methods.

Student Perceived Effectiveness of Computer Technology Use in Post-Secondary Classrooms.

Abstract This study investigated the relationship between the amount of computer technology used in post-secondary education courses, students’ perceived effectiveness of technology use, and global course evaluations. Survey data were collected from 922 students in 51 courses at both the graduate and undergraduate levels. The survey consisted of 65 items broken down into seven areas, namely: (1) student characteristics, (2) learning experiences and course evaluations, (3) learning strategies, (4) instructional techniques, (5) computer use in course, (6) perceived effectiveness of computer use and (7) personal computer use. Contrary to expectations, no significant relationship was found between computer use and global course evaluations, nor was there a relationship between perceived effectiveness of computer use and global course evaluations. However, the results did yield a positive relationship between global course evaluations and the learning experiences that students engaged in. Students also indicated that they valued the use of computer technology for learning. Descriptive statistics on questions related to personal computer use show a strong favorable response to computer use and: facilitation of learning, value-added aspects such as usefulness to other classes and/or career, learning material in a more meaningful way, and working in groups with other students.

The effects of technology use in postsecondary education

This meta-analysis is a study of the experimental literature of technology use in postsecondary education from 1990 up to 2010 exclusive of studies of online or distance education previously reviewed by Bernard et al. (2004). It reports the overall weighted average effects of technology use on achievement and attitude outcomes and explores moderator variables in an attempt to explain how technology treatments lead to positive or negative effects. Out of an initial pool of 11,957 study abstracts, 1105 were chosen for analysis, yielding 879 achievement and 181 attitude effect sizes after pre-experimental designs and studies with obvious methodological confounds were removed. The random effects weighted average effect size for achievement was g+ = 0.27, k = 879, p < .05, and for attitude outcomes it was g+ = 0.20, k = 181, p < .05. The collection of achievement outcomes was divided into two sub-collections, according to the amount of technology integration in the control condition. These were no technology in the control condition (k = 479) and some technology in the control condition (k = 400). Random effects multiple meta-regression analysis was run on each sub-collection revealing three significant predictors (subject matter, degree of difference in technology use between the treatment and the control and pedagogical uses of technology). The set of predictors for each sub-collection was both significant and homogeneous. Differences were found among the levels of all three moderators, but particularly in favor of cognitive support applications. There were no significant predictors for attitude outcomes.

Supporting the Learning Process with Collaborative Concept Mapping Using Computer-based Communication Tools and Processes

University students are too often challenged by their limited skills in application, investigation, relational thinking, and communication of ideas. In this study, we have combined 3 tools that potentially can support and foster students’ development in the above mentioned areas through student collaboration, concept mapping, and electronic technologies. The participants in this study were 26 students in two intact classes in learning theories. In groups of 3 to 5 students, they were asked to generate 3 concept maps and accompanying prose over the term on 3 major issues in the field of learning. Through the use of interviews, questionnaires, and student generated concept maps, students reportedly enjoyed concept mapping for its organizational and relational properties but preferred sharing their concept maps and dialoguing with one another in a synchronous mode where immediate feedback and flow of thinking could be maintained when involved in constructing maps. Moreover, they did not like the redundancy offered by both prose and concept map outputs, suggesting that while concept mapping can be an arena for generating and generally structuring ideas, prose can be a means of communicating such ideas in a form that is common to most people. This is particularly important for teachers and students who have difficulty navigating through maps alone.

A Human-Computer Partnership: The Tutor/Child/Computer Triangle Promoting the Acquisition of Early Literacy Skills

Abstract This study involved the analysis of the complex interactions that take place between tutors and preschool children using a computer during early literacy tutoring sessions. Eight five-year-old pre- and early-readers attending a childcare centre participated in daily 20-minute tutoring sessions for two weeks. The literacy software (a beta version) was especially designed to guide tutors while working one-on-one with elementary school students falling into the lower 30% of reading achievement (i.e., at-risk). Parent surveys, videotaped tutor/child sessions, independent observer data, and tutor reports yielded rich descriptions of the tutor/child/computer process. Rigorous grounded theory analyses generated three comprehensive themes: rapport, motivation, and scaffolding. The first focused on interpersonal issues, the latter two on teaching/learning. Implications for practice are discussed.

Are We Using Technology for Learning

This study investigated the role that computer technology plays in transforming the learning process in higher education. Specifically, we looked at the relationship between computer-technology use, active learning, and perceived course effectiveness. The sample consisted of 1966 students in 81 graduate and undergraduate classes at a large, urban university. The survey categories were: 1) learner preferences; 2) course structure; 3) active learning; 4) time on task; 5) learning with technology; 6) perceived effectiveness of computer use; 7) context of computer use; and 8) overall perceived course effectiveness. Results suggest that there is a relationship between computer technology, active learning, and perceived course effectiveness. Students who use computer technology a lot appear to benefit the most from active learning.

Technology-supported student interaction in post-secondary education

The present study extends the results of a larger meta-analysis that addressed the effects of technology use on student achievement and attitudes in postsecondary education. The focus of the current meta-analysis is the use of technology to enable instructional conditions that promote collaborative interactions among learners. More specifically, it aims to compare the impact of designed interaction treatments (i.e., collaborative activities intentionally built into course design) and contextual interaction treatments (i.e., course conditions that result in high levels of student-student interaction but are not intentionally designed to promote collaboration) on student learning outcomes. Results indicate that designed treatments outperform contextual treatments ( g ? ?=?0.52, k?=?25 vs. g ? ?=?0.11, k?=?20, QBetween?=?7.91, p?<?.02) on measures of achievement, emphasizing the importance of planning and instructional design in technology integration in postsecondary education. The results are discussed in relation to the literature of student-student interaction and collaborative learning. Meta-analysis of designed and contextual interaction treatments.Covers studies published from 1990 up to 2010.Designed (k?=?25) versus contextual interaction treatments (k?=?20).Comparison was significant ( g ? ?=?0.52, vs. g ? ?=?0.11, QBetween?=?7.91, p?<?.02).Moderator variable analysis yielded positive outcomes for cognitive support tools.

Evaluation of the Redesign of an Undergraduate Cell Biology Course

This article offers a case study of the evaluation of a redesigned and redeveloped laboratory-based cell biology course. The course was a compulsory element of the biology program, but the laboratory had become outdated and was inadequately equipped. With the support of a faculty-based teaching improvement project, the teaching team redesigned the course and re-equipped the laboratory, using a more learner-centered, constructivist approach. The focus of the article is on the project-supported evaluation of the redesign rather than the redesign per se. The evaluation involved aspects well beyond standard course assessments, including the gathering of self-reported data from the students concerning both the laboratory component and the technical skills associated with the course. The comparison of pre- and postdata gave valuable information to the teaching team on course design issues and skill acquisition. It is argued that the evaluation process was an effective use of the scarce resources of the teaching improvement project.