Gregory K. W. K. Chung

The use of computer-based collaborative knowledge mapping to measure team processes and team outcomes

In this study we examined the feasibility and validity of using a computer-based, networked collaborative knowledge mapping system to measure teamwork skills. Student groups were assessed with our system twice in the same academic year, once in the fall and once in the following spring. Our study focused on the nature of the interaction between team members as they jointly constructed a knowledge map. Each student was randomly assigned to a team and communicated (anonymously) with other members by sending pre-defined messages. Teamwork processes were measured by examining message usage. Each message was categorized as belonging to one of six team processes: (a) adaptability, (b) communication, (c) coordination, (d) decision-making, (e) interpersonal, and (f) leadership. Team performance was measured by scoring each team’s knowledge map using four expert maps as the criterion. No significant correlation was found between the team processes and team outcomes. This unexpected finding may be due in part to a split-attention effect resulting from the design of the user interface. However, student teams were able to successfully construct knowledge maps, suggesting that our general approach to using networked computers to measure group processes remain viable given existing alternatives.

Issues in the Computer-based Assessment of Collaborative Problem Solving

Abstract Collaborative problem solving skills are considered necessary skills for success in todays world of work and school. Collaborative learning refers to learning environments in which small groups of people work together to achieve a common goal, and problem solving is using cognitive processing to find a solution to a problem. Thus, collaborative problem solving is defined as problem solving activities that involve interactions among a group of individuals. This paper will address several key issues (e.g. theory and measurement of collaborative problem solving and issues in measuring problem solving processes). We rely on computerisation of the administration, scoring and reporting of collaborative problem solving skills, thus potentially increasing reliability and validity.

New Directions in Rifle Marksmanship Research

We propose that future rifle marksmanship research be framed within a phases-of-skill-development model (Ackerman, 1987, 1992; Anderson, 1982; Fitts & Posner, 1967). Prior research on predicting shooting performance suggests a deceptively complex task sensitive to variations in the individual, equipment, and environment. We argue that rifle marksmanship research should be framed around perceptual-motor, cognitive, affective, equipment, and environmental variables. Although it is unlikely that equipment and environment can be controlled, much can be learned—with training implications—about how perceptual-motor, cognitive, and affective variables relate to shooting performance. The phases-of-skill-development model is silent on affective variables but suggests that cognitive factors will be most sensitive to individuals learning how to shoot, and perceptual-motor variables most sensitive to individuals who already know how to shoot. Identification of where trainees are in their skill development could lead to more efficient and targeted training and decreased training costs.

Validating Measures of Algebra Teacher Subject Matter Knowledge and Pedagogical Content Knowledge

The purpose of this study was to validate inferences about scores of one task designed to measure subject matter knowledge and three tasks designed to measure aspects of pedagogical content knowledge. Evidence for the validity of inferences was based on two expectations. First, if tasks were sensitive to expertise, we would find group differences. Second, tasks that measured similar types of knowledge would correlate strongly, and tasks that measured different types of knowledge would correlate weakly. We recruited and assessed 4 groups of participants including 46 experienced algebra teachers (2+ years experience), 17 novice algebra teachers (0–2 years experience), 10 teaching experts, and 13 subject matter experts. Results indicate that one task differentiated among levels of expertise and measured several aspects of knowledge needed to teach algebra. Results also highlight that future studies should use a combination of tasks to accurately measure different aspects of teacher knowledge.

Guidelines for the Design and Implementation of Game Telemetry for Serious Games Analytics

The design of game telemetry requires careful attention to the chain of reasoning that connects low-level behavioral events to inferences about players’ learning and performance. Measuring performance in serious games is often difficult because seldom do direct measures of the desired outcome exist in the game. Game telemetry is conceived as the fundamental element from which measures of player performance are developed. General psychometric issues are raised for game-based measurement, and data issues are raised around format, context, and increasing the meaningfulness of the data itself. Practical guidelines for the design of game telemetry are presented, including targeting in-game behaviors that reflect cognitive demands, recoding data at the finest usable grain size, representing the data in a format usable by the largest number of people, and recording descriptions of behavior and not inferences with as much contextual information as practical. A case study is presented on deriving measures in a serious game intended to teach fraction concepts.

Peak Performance Trainer (PPTTM): Interactive Neuro-educational Technology to Increase the Pace and Efficiency of Rifle Marksmanship Training

Marksmanship training involves a combination of classroom instructional learning and field practice involving the instantiation of a well-defined set of sensory, motor and cognitive skills. Current training procedures rely heavily on conventional classroom instruction often with qualitative assessment based on observation (i.e. coaching). We have developed a novel device called the Peak Performance Trainer (PPTTM) which can accelerate the progression from novice-to-expert based on automated inferences from neurophysiological measurements. Our previous work has revealed specific EEG correlates to stages of skill acquisition in simple learning and memory tasks. We have incorporated this knowledge as well as an array of other physiological metrics to develop a field-deployable training technology with continuous physiological monitoring in combination with simultaneous measures of performance, workload, engagement and distraction, accuracy, speed and efficiency. This paper outlines the features of the PPT and the preliminary results of its use in marksmanship training.

Supporting children’s progress through the PBS KIDS learning analytics platform

Abstract A 2010–2015 Ready to Learn grant from the US Department of Education supported the research, development, and dissemination of PBS KIDS math and literacy content for kids aged 2–8, and new ways to measure and report on children’s learning. PBS KIDS, in collaboration with UCLA CRESST, developed a prototype learning analytics platform (LAP) to analyze back-end data from children’s interactions with PBS KIDS transmedia content, and provide custom reports to parents that enable them to better identify and support their child’s individual learning needs. Evidence suggests that the PBS KIDS prototype LAP demonstrates potential for assessment and prediction of mathematics performance, including on the TEMA-3 standardized assessment for early mathematics. Similarly, early research findings suggest that the reporting applications developed to provide LAP information are viewed favorably by parents and promote interest in learning more about, monitoring, and engaging with their kids around learning. Implications for the future are discussed.

First Evaluation of the Physics Instantiation of a Problem-Solving-Based Online Learning Platform

Problem solving is a commonly used learning activity around which a large number of state-of-the-art Intelligent Tutoring Systems are developed and evaluated. In this paper, we present our problem-solving-based online learning platform and discuss a preliminary laboratory trial of this platform. While the platform itself is domain independent, for this evaluation, it was instantiated with a collection of problems from the unit of Electricity and Magnetism taught in high-school-level physics. Results indicate pedagogical effectiveness of problem solving in the Physics instance of the platform, with 41% of participants exceeding the stringent reliable change index.

Assessment Gaze, Refraction, and Blur: The Course of Achievement Testing in the Past 100 Years

This chapter addresses assessment (testing) with an emphasis on the 100-year period since the American Education Research Association was formed. The authors start with definitions and explanations of contemporary tests. They then look backward into the 19th century to significant work by Horace Mann and Herbert Spencer, who engendered two parallel purposes for assessment: evaluating effects of education and identifying individual differences. The authors consider the interplay of these orientations over the years. After discussing policy impacts on assessment, they discuss the evolution of the concept of validity as it relates to changing rationales for testing. To enrich the reader’s comprehension, the authors also discuss perspectives on innovation in technology and in quantitative analysis. They conclude with questions that summarize current concerns with assessment. Finally, they consider future prospects for assessment. They foresee the continued convergence of assessment purposes, innovative learning technology, and new psychometric challenges.

I-NET ® : Interactive neuro-educational technology to accelerate skill learning

The learning of a novel task currently rely heavily on conventional classroom instruction with qualitative assessment and observation. Introduction of individualized tutorials with integrated neuroscience-based evaluation techniques could significantly accelerate skill acquisition and provide quantitative evidence of successful training. We have created a suite of adaptive and interactive neuro-educational technologies (I-NET) to increase the pace and efficiency of skill learning. It covers four major themes: 1) Integration of brain monitoring into paced instructional tutorials, 2) Identifying psychophysiological characteristics of expertise using a model population, 3) Developing sensor-based feedback to accelerate novice-to-expert transition, 4) Identifying neurocognitive factors that are predictive of skill acquisition to allow early triage and interventions. We selected rifle marksmanship training as the field of application. Rifle marksmanship is a core skill for the Army and Marine Corps and it involves a combination of classroom instructional learning and field practice involving instantiation of a well-defined set of sensory, motor and cognitive skills. The instrumentation that incorporates the I-NET technologies is called the Adaptive Peak Performance Trainer (APPT®). Preliminary analysis of pilot study data for performance data from a novice population that used this device revealed an improved learning trajectory.