Lee Dee Miller

Lessons Learned from Comprehensive Deployments of Multiagent CSCL Applications I-MINDS and ClassroomWiki

Recent years have seen a surge in the use of intelligent computer-supported collaborative learning (CSCL) tools for improving student learning in traditional classrooms. However, adopting such a CSCL tool in a classroom still requires the teacher to develop (or decide on which to adopt) the CSCL tool and the CSCL script, design the relevant pedagogical aspects (i.e., the learning objectives, assessment method, etc.) to overcome the associated challenges (e.g., free riding, student assessment, forming student groups that improve student learning, etc). We have used a multiagent-based system to develop a CSCL application and multiagent-frameworks to form student groups that improve student collaborative learning. In this paper, we describe the contexts of our three generations of CSCL applications (i.e., I-MINDS and ClassroomWiki) and provide a set of lessons learned from our deployments in terms of the script, tool, and pedagogical aspects of using CSCL. We believe that our lessons would allow 1) the instructors and students to use intelligent CSCL applications more effectively and efficiently, and help to improve the design of such systems, and 2) the researchers to gain additional insights into the impact of collaborative learning theories when they are applied to real-world classrooms.

Evaluating the use of learning objects in CS1

Learning objects (LOs) have been previously used in computer science education. However, analyses in previous studies have been limited to surveys with limited numbers of LOs and students. The lack of copious quantitative data on how LOs impact student learning makes detailed analysis of LO usefulness problematic. Using an empirical approach, we have studied a suite of LOs, comprehensive in both the content covered and the range of difficulty, deployed to CS1 courses from 2007-2010. We review previous work on predictors of achievement and impact of active learning and feedback. We also provide a high-level overview of our LO deployment. Finally, based on our analysis of student interaction data, we found that (1) students using LOs have significantly higher assessment scores than the control group, (2) several student attributes are significant predictors of learning, (3) active learning has a significant effect on student assessment scores, and (4) feedback does not have a significant effect, but there are variables with significant moderating effects.

Empirical usage metadata in learning objects

The iLOG Project (Intelligent Learning Object Guide) is designed to augment multimedia learning objects with information about (1) how a learning object has been used, (2) how it has impacted instruction and learning, and (3) how it should be used. The goal of the project is to generate metadata tags from data collected while students interact with learning objects; these metadata tags can then be used to help teachers identify learning objects that match the educational and experiential backgrounds of their students. The project involves the development of an agent-based intelligent system for tracking student interaction with learning objects, in tandem with an extensive learning research agenda. This paper provides an overview of this NSF-funded project, focusing on the instructional approach and research on varying levels of active learning and feedback. Using a randomized design and a hierarchical linear modeling framework, research showed that the active learning conditions resulted in significantly higher student learning. The elaborative feedback results approached (p = .056), but did not reach, the established significance criteria of alpha = .05. Both active learning conditions and one of the elaborative feedback conditions resulted in significantly higher content assessment scores compared to a control group.

Teaching Using Computer Games

Computer games have long been used for teaching. Current reviews lack categorization and analysis using learning models which would help instructors assess the usefulness of computer games. We divide the use of games into two classes: game playing and game development. We discuss the Input-Process-Outcome (IPO) model for the learning process when playing computer games. We also propose a new Input-Process-Outcome model for explaining the learning through game development (IPO-GD). Using both learning models, we review recent uses of computer games. Based on our review, we recommend: 1) using the IPO model when selecting games; 2) using the IPO-GD model for game development; and 3) creating support repositories for related curriculum material.

Case-based learning mechanisms to deliver learning materials

In this paper, we discuss an integrated framework of case-based learning (CBL) in an agent that intelligently delivers learning materials to students. The agent customizes its delivery strategy for each student based on the students background profile and his or her interactions with the graphic user interface (GUI) to our system, and based on the usage history of the learning materials. The agents decision-making process is powered by case-based reasoning (CBR). To improve its reasoning process, our agent learns the differences between good cases (cases with a good solution for its problem space) and bad cases (cases with a bad solution for its problem space). It also meta-learns adaptation heuristics, the significance of input features of the cases, and the weights of a content graph for symbolic feature values. We have also built a simulation to comprehensively test the learning behavior of our agent.

Revising computer science learning objects from learner interaction data

Learning objects (LO) have previously been used to help deliver introductory computer science (CS) courses to students. Students in such introductory CS courses have diverse backgrounds and characteristics requiring revision to LO content and assessment to promote learning in all students. However, revising LOs in an ad hoc manner could make student learning harder for subsequent deployments. To address this problem, we present a systematic revision process for LOs (LOSRP) using proven techniques from educational research including Blooms Taxonomy levels, item-total correlation, and Cronbachs Alpha. LOSRP uses these validation methods to answer seven questions in order to diagnose what needs to be revised in the LO. Then, LOSRP provides guidelines on revising LOs for each of the seven questions. As an example, we discuss how LOSRP was used to revise the content and assessment for 16 LOs deployed to over 400 students in introductory CS courses in 2009. Lastly, although initially designed for LO revision, we briefly discuss how LOSRP could be used for assessment revision in intelligent tutoring systems.

iLOG: a framework for automatic annotation of learning objects with empirical usage metadata

Learning objects (LOs) are digital or non-digital entities used for learning, education or training commonly stored in repositories searchable by their associated metadata. Unfortunately, based on the current standards, such metadata is often missing or incorrectly entered making search difficult or impossible. In this paper, we investigate automating metadata generation for SCORM-complaint LOs based on user interactions with the LO and static information about LOs and users. Our framework, called the Intelligent Learning Object Guide (iLOG), involves real-time tracking of each user sessions (an LO Wrapper), offline data mining to identify key attributes or patterns on how the LOs have been used as well as characteristics of the users (MetaGen), and the selection of these findings as metadata. Mechanisms used in the data mining include data imputation via clustering, association rule mining, and feature selection ensemble. This paper describes the methodology of automatic annotation, presents the results on the evaluation and validation of the algorithms, and discusses the resulting metadata. We have deployed our iLOG implementation for five LOs in introductory computer science topics and collected data for over 1400 sessions. We demonstrate that iLOG successfully tracks user interactions that can be used to automate the generation of meaningful empirical usage metadata for different stakeholder groups including learners and instructors, LO developers, and researchers.

ILMDA: an intelligent learning materials delivery agent and simulation

In this paper, we describe an intelligent agent that delivers learning materials adaptively to different students, factoring in the usage history of the learning materials, the student static background profile, and the student dynamic activity profile. Our assumption is that through the interaction of a student going through a learning material (i.e., a topical tutorial, a set of examples, and a set of problems), an agent will be able to capture and utilize the students activity as the primer to select the appropriate example or problem to administer to the student. In addition, our agent monitors the usage history of the learning materials and derives empirical observations that improve its performance. We have built an end-to-end infrastructure, with a GUI front-end, an agent powered by case-based reasoning, and a multi-database backend. Preliminary experiments based on a comprehensive simulator show the feasibility, correctness, and learning capability of our methodology and system