Laura K. Varner

Using Automated Indices of Cohesion to Evaluate an Intelligent Tutoring System and an Automated Writing Evaluation System

We present an evaluation of the Writing Pal (W-Pal) intelligent tutoring system (ITS) and the W-Pal automated writing evaluation (AWE) system through the use ofcomputational indices related to text cohesion. Sixty-four students participated in this study. Each student was assigned to either the W-Pal ITS condition or the W-Pal AWE condition. The W-Pal ITS includes strategy instruction, game-based practice, and essay-based practice with automated feedback. In the ITS condition, students received strategy training and wrote and revised one essay in each of the 8 training sessions. In the AWE condition, students only interacted with the essay writing and feedback tools. These students wrote and revised two essays in each of the 8 sessions. Indices of local and global cohesion reported by the computational tools Coh-Metrix and the Writing Assessment Tool (WAT) were used to investigate pretest and posttest writing gains. For both the ITS and the AWE systems, training led to the increased use of global cohesion features in essay writing. This study demonstrates that automated indices of text cohesion can be used to evaluate the effects of ITSs and AWE systems and further demonstrates how text cohesion develops as a result of instruction, writing, and automated feedback.

Expectations of Technology: A Factor to Consider in Game-Based Learning Environments

This study investigates how students’ prior expectations of technology affect overall learning outcomes across two adaptive systems, one game-based (iSTART-ME) and one non-game based (iSTART-Regular). The current study (n=83) is part of a larger study (n=124) intended to teach reading comprehension strategies to high school students. Results revealed that students’ prior expectations impacted learning outcomes, but only for students who had engaged in the game-based system. Students who reported positive expectations of computer helpfulness at pretest showed significantly higher learning outcomes in the game-based system compared to students who had low expectations of computer helpfulness. The authors discuss how the incorporation of game-based features in an adaptive system may negatively impact the learning outcomes of students with low technology expectations.

The Impact of System Interactions on Motivation and Performance in a Game-Based Learning Environment

The current study examined how students’ frequency of interactions with game-based features impacted their system performance (i.e., total trophies won and achievement levels earned) and attitudes toward the game-based system, iSTART-ME. This study (n=40) was a part of a larger study (n=124) conducted with high school students. Results indicate that students’ interactions with game-based features were positively related to both their system performance and their posttest attitudes toward the system. These findings provide further support showing that the integration of game-based features has positive effects on students within educational learning environments.

Linguistic Content Analysis as a Tool for Improving Adaptive Instruction

This study investigates methods to automatically assess the features of content texts within an intelligent tutoring system (ITS). Coh-Metrix was used to calculate linguistic indices for texts (n = 66) within the reading strategy ITS, iSTART. Coh-Metrix indices for the system texts were compared to students’ (n = 126) self-explanation scores to examine the degree to which linguistic indices predicted students’ self-explanation quality. Initial analyses indicated no relation between self-explanation scores on a given text and its linguistic properties. However, subsequent analyses indicated the presence of robust text effects when analyses were separated for high and low reading ability students.