John C. Nesbit

Learning With Concept and Knowledge Maps: A Meta-Analysis

This meta-analysis reviews experimental and quasi-experimental studies in which students learned by constructing, modifying, or viewing node-link diagrams. Following an exhaustive search for studies meeting specified design criteria, 67 standardized mean difference effect sizes were extracted from 55 studies involving 5,818 participants. Students at levels ranging from Grade 4 to postsecondary used concept maps to learn in domains such as science, psychology, statistics, and nursing. Posttests measured recall and transfer. Across several instructional conditions, settings, and methodological features, the use of concept maps was associated with increased knowledge retention. Mean effect sizes varied from small to large depending on how concept maps were used and on the type of comparison treatment. Significant heterogeneity was found in most subsets.

LEARNING OBJECT EVALUATION: COMPUTER-MEDIATED COLLABORATION AND INTER-RATER RELIABILITY

Abstract Learning objects offer increased ability to share learning resources so that system-wide production costs can be reduced. But how can users select from a set of similar learning objects in a repository and be assured of quality? This article reviews recent developments in the establishment of learning object repositories and metadata standards, and presents a formative reliability analysis of an online, collaborative method for evaluating quality of learning objects. The method uses a 10-item Learning Object Review Instrument (LORI) within a Convergent Participation evaluation model that brings together instructional designers, media developers, and instructors. The inter-rater reliability analysis of 12 raters evaluating eight learning objects identified specific items in LORI that require further development. Overall, the collaborative process substantially increased the reliability and validity of aggregate learning object ratings. The study concludes with specific recommendations including changes to LORI items, a rater training process, and requirements for selecting an evaluation team.

A Convergent Participation Model for Evaluation of Learning Objects

The properties that distinguish learning objects from other forms of educational software - global accessibility, metadata standards, finer granularity and reusability - have implications for evaluation. This article proposes a convergent participation model for learning object evaluation in which representatives from stakeholder groups (e.g., students, instructors, subject matter experts, instructional designers, and media developers) converge toward more similar descriptions and ratings through a two-stage process supported by online collaboration tools. The article reviews evaluation models that have been applied to educational software and media, considers models for gathering and meta-evaluating individual user reviews that have recently emerged on the Web, and describes the peer review model adopted for the MERLOT repository. The convergent participation model is assessed in relation to other models and with respect to its support for eight goals of learning object evaluation: (1) aid for searching and selecting, (2) guidance for use, (3) formative evaluation, (4) influence on design practices, (5) professional development and student learning, (6) community building, (7) social recognition, and (8) economic exchange.

A Cross-Cultural Examination of the Psychometric Properties of Responses to the Achievement Goal Questionnaire.

The psychometric properties of scores from the Achievement Goal Questionnaire were examined in samples of Japanese (N = 326) and Canadian (N = 307) postsecondary students. Previous research found evidence of a four-factor structure of achievement goals in U.S. samples. Using confirmatory factor-analytic techniques, the authors found strong evidence for the four-factor structure of achievement goals in both the Canadian and Japanese populations. Subsequent multigroup structural equation modeling indicated the metric invariance of this four-factor structure across the two populations.

Rating learning object quality with distributed Bayesian belief networks: the why and the how

As differing evaluation instruments are adopted in learning object repositories serving specialized communities of users, what methods can be adopted for translating evaluative data across instruments in order to share this data among different repositories? How can evaluation from different reviewers be properly integrated? How can explicit and implicit measures of preference and quality be combined to recommend objects to users? In this research we studied the application of Bayesian belief network (BBN) to the problem of insufficient and incomplete reviews during learning objects evaluation, and translating and integrating data among different quality evaluation instruments and measures. Two BBNs were constructed to probabilistically model relationships among different roles of reviewers as well as among items of different evaluation measurements. Initial testing using hypothetic data showed that the model was able to make potentially useful inferences about different dimensions of learning object quality. We further extend our model over geographic distances assuming that the reviewers would be distributed and that each reviewer would change the underlying BBN network (to a certain extent) to suit his/her expertise. We highlight issues that arise due to a highly distributed and personalized BBN network that can be used to make valid inferences about learning object quality.

Learning from Animated Concept Maps with Concurrent Audio Narration

An animated concept map is a presentation of a network diagram in which nodes and links are sequentially added or modified. An experiment compared learning from animated concept maps and text by randomly assigning 133 undergraduates to study 1 of 4 narrated animations presenting semantically equivalent information accompanied by identical audio narration. Two of the animations presented text; one with concurrent audio and another with delayed audio. Two of the animations presented concept maps; one in black and white and the other with nodes colored to represent semantic relatedness. The concept map groups outperformed the text groups on free recall (p < .05). The black-and-white concept map group outperformed the text groups on a multiple-choice knowledge test (p < .05). No advantages were statistically detected for color enhancements of the animated map. The results indicate that verbal information can be effectively communicated by learner-paced animated concept maps accompanied by audio narrations.

Evaluating Learning Objects Across Boundaries: The Semantics of Localization

ABSTRACT Learning object repositories and evaluation tools have the potential to serve as sites for interaction among different cultures and communities of practice. This paper outlines web-based learning object evaluation tools we have developed, describes our current efforts to extend those tools to a wider range of user communities, and considers methods for fostering interaction among user communities. We discuss the recommendation of objects across community boundaries, and approaches to mapping between languages, ontologies and work practices. Keywords. e-learning, reviews, quality, repositories, communities, eLera, LORI, convergent participation, collaboration, professional development THE NEED FOR EVALUATION ACROSS COMMUNITIES Perhaps the main feature distinguishing learning objects from other educational software is their ready availability through web-based repositories, collections that can be searched with standardized metadata. Because user communities such as elementary school teachers and university instructors have different requirements and expectations, specialized repositories are emerging that are interlinked by metadata and interoperability standards. Within the next five years, the U.S. National Science Digital Library (NSDL) is predicted to grow to include as many as 100,000 collections representing over a million learning objects (Saylor, 2002). We believe that the effectiveness of most online learning resources is severely limited because they do not conform to design principles established by research, and have not been subjected to formative user testing. Better education of instructional designers, teachers and other e-learning professionals is part of the solution. And there is an immediate need for methods and tools to facilitate the development, dissemination and retrieval of high quality resources. The scope of the quality problem is such that a variety of specialized evaluation methods is required. For example, teachers and learners may only be induced to evaluate if the instruments are uncomplicated and demand very little additional time and effort. In

How Effective are Intelligent Tutoring Systems in Computer Science Education

A meta-analysis on the effectiveness of Intelligent Tutoring Systems (ITS) in computer science education compared the learning outcomes of ITS and non-ITS instruction. A search of the literature found 22 effect sizes (involving 1,447 participants) that met the pre-defined inclusion criteria. Although most of the ITS were used to teach programming, other topics such as database design and computer literacy were also represented. There was a significant overall effect size favoring the use of ITS. There was a significant advantage of ITS over teacher-led classroom instruction and non-ITS computer-based instruction. ITS were more effective than the instructional methods to which they were compared regardless of whether they modeled misconceptions and regardless of whether they were the primary means of instruction or were an integrated component of learning activities that included other means of instruction.

Self-Regulated Learning, Motivation and Goal Theory: Implications for Instructional Design and E-Learning

Instructional designers can use evidence-based theories of motivation and self-regulated learning to construct e-learning environments that promote goal-setting and achievement. We have used gStudy, educational software designed according to principles of self-regulated learning, to construct a Goal-Setting Kit. The Goal-Setting Kit incorporates tests, scaffolds, concept maps, and exercises that allow students to express, organize, and monitor their strategies for achieving personal and instructional goals

Quality Rating and Recommendation of Learning Objects

The unceasing growth of the Internet has led to new modes of learning in which learners routinely interact on-line with instructors, other students, and digital resources. Much recent research has focused on building infrastructure for these activities, especially to facilitate searching, filtering, and recommending on-line resources known as learning objects. Although newly defined standards for learning object metadata are expected to greatly improve searching and filtering capabilities, learners, instructors, and instructional developers may still be faced with choosing from many pages of object listings returned from a single learning object query. The listed objects tend to vary widely in quality. With current metadata and search methods, those who search for learning objects waste time and effort groping through overwhelming masses of information, often finding only poorly designed and developed instructional materials. Hence, there is a clear need for quality evaluations prior to making a recommendation that can be communicated in a coherent, standardized format to measure the quality of learning objects.