Kirsten R. Butcher

Automatically characterizing resource quality for educational digital libraries

With the rise of community-generated web content, the need for automatic characterization of resource quality has grown, particularly in the realm of educational digital libraries. We demonstrate how identifying concrete factors of quality for web-based educational resources can make machine learning approaches to automating quality characterization tractable. Using data from several previous studies of quality, we gathered a set of key dimensions and indicators of quality that were commonly identified by educators. We then performed a mixed-method study of digital library curation experts, showing that our characterization of quality captured the subjective processes used by the experts when assessing resource quality for classroom use. Using key indicators of quality selected from a statistical analysis of our expert study data, we developed a set of annotation guidelines and annotated a corpus of 1000 digital resources for the presence or absence of these key quality indicators. Agreement among annotators was high, and initial machine learning models trained from this corpus were able to identify some indicators of quality with as much as an 18% improvement over the baseline.

Computational foundations for personalizing instruction with digital libraries

This paper describes our progress towards automating adaptive personalized instruction based on student conceptual understandings using digital libraries. The reported approach merges conversational learning theory with advances in natural language processing to enable personalized pedagogical interactions. Multi-document summarization techniques serve as the computational basis to process digital library resources and automatically construct a rich domain model on earthquakes and plate tectonics for high school age learners. Shallow semantic analysis and graph-based techniques are used to computationally diagnose student understandings that enable conceptual personalizations integrating digital library resources. The evaluation of the implemented algorithms indicates that digital libraries may serve as knowledge platforms to support the automated construction of rich domain models and the diagnosis of student conceptual understandings. Furthermore, this approach introduces a novel and effective alternative to prior work in adaptive learning environments in terms of scalability and portability, thus tackling important challenges associated with supporting personalized instruction using digital libraries.

Support of content and rhetorical processes of writing: Effects on the writing process and the written product

The effects of content and rhetorical prompts on writing process activities and the quality of written products were examined. We also examined the usefulness of latent semantic analysis (LSA; Landauer & Dumais, 1997)-a computational technique for representing the content of documents-as a tool for assessing texts. Participants used varied combinations of prompts designed to support content and rhetorical processes. In Experiment 1, content and rhetorical processes were supported only during composition. In Experiment 2, content and rhetorical processes were supported during domain learning and writing. Time spent in 3 writing activities (planning, drafting, revising) was measured, and professional writing instructors and LSA assessed text quality. Content prompts extended time spent writing and were related to improved text quality; rhetorical prompts demonstrated some influence on planning and global text quality only when presented during domain learning. In both experiments, LSA generated consistent judgments of writing quality that resembled human grading.

Introductory Circuit Analysis Learning From Abstract and Contextualized Circuit Representations: Effects of Diagram Labels

Novice learners are typically unfamiliar with abstract engineering symbols. They are also often unaccustomed to instructional materials consisting of a combination of text, diagrams, and equations. This raises the question of whether instruction on elementary electrical circuit analysis for novice learners should employ contextualized representations of the circuits with familiar components, such as batteries, or employ abstract representations with the abstract engineering terms and symbols. A further question is if text labels in the circuit diagrams would aid these learners. This study examined these research questions with a “2 × 3” experimental design, in which the two forms of representation (abstract or contextualized) were considered under three types of diagram labeling (no labels, static labels, or interactive labels). The design was implemented in an instructional module on elementary circuit analysis for novice learners. Results indicated that abstract representations led to higher near- and far-transfer post-test scores, and that interactive (student-generated) labeling resulted in higher near-transfer scores than either the no-labels or static-labels conditions. These findings suggest that abstract representations promote the development of deep, transferrable knowledge and that generative methods of integration, such as interactive diagram labeling, can facilitate learning with multiple external representations.

Characterizing and Predicting the Multifaceted Nature of Quality in Educational Web Resources

Efficient learning from Web resources can depend on accurately assessing the quality of each resource. We present a methodology for developing computational models of quality that can assist users in assessing Web resources. The methodology consists of four steps: 1) a meta-analysis of previous studies to decompose quality into high-level dimensions and low-level indicators, 2) an expert study to identify the key low-level indicators of quality in the target domain, 3) human annotation to provide a collection of example resources where the presence or absence of quality indicators has been tagged, and 4) training of a machine learning model to predict quality indicators based on content and link features of Web resources. We find that quality is a multifaceted construct, with different aspects that may be important to different users at different times. We show that machine learning models can predict this multifaceted nature of quality, both in the context of aiding curators as they evaluate resources submitted to digital libraries, and in the context of aiding teachers as they develop online educational resources. Finally, we demonstrate how computational models of quality can be provided as a service, and embedded into applications such as Web search.

Towards automatic conceptual personalization tools

This paper describes the results of a study designed to validate the use of domain competency models to diagnose student scientific misconceptions and to generate personalized instruction plans using digital libraries. Digital library resources provided the content base for human experts to construct a domain competency model for earthquakes and plate tectonics encoded as a knowledge map. The experts then assessed student essays using comparisons against the constructed domain competency model and prepared personalized instruction plans using the competency model and digital library resources. The results from this study indicate that domain competency models generated from select digital library resources may provide the desired degree of content coverage to support both automated diagnosis and personalized instruction in the context of nationally-recognized science learning goals. These findings serve to inform the design of personalized instruction tools for digital libraries.

Learning from abstract and contextualized representations: The effect of verbal guidance

An experiment examined the effects of providing explicit verbal guidance to learners in integrating information with abstract or contextualized representations during computer-based learning of engineering. Verbal guidance supported learners in identifying correspondences and making mental connections among multiple textual and diagrammatic representations. Results from a 2 (abstract (A) or contextualized (C) representation)x2 (no guidance or guidance) design showed that without guidance, abstract representations led to better transfer than contextualized representations. Moreover, learners in the contextualized representation group benefitted from the guidance, while the abstract representation group did not benefit from guidance. These findings suggest that abstract representations promote the development of deep, transferrable knowledge and that verbal guidance denoting correspondences among representations can facilitate learning when less effective representational formats are utilized.

Do graphical search interfaces support effective search for and evaluation of digital library resources

This paper explores the cognitive processes and online behaviors in which preservice teachers engage when seeking educational resources for classroom instruction. Participants used graphical and keyword search interfaces provided by a large-scale digital library (NSDL.org) and a keyword search interface from a large, commercial search engine (Google.com) to complete searches for online materials that would support classroom instruction. Overall, findings from the current work indicate that a graphical search interface can support comprehension by providing a conceptual organization of domain content during digital search and evaluation. Findings also show that digital libraries allow users to offload processing related to resource trustworthiness, thereby increasing cognitive capacity for other purposes.

Learning with Diagrams: Effects on Inferences and the Integration of Information

Students studied materials about the human heart and circulatory system using either (a) text only, (b) text with simple diagrams, or (c) text with detailed diagrams. During learning, students self-explained [1] the materials. Explanations were transcribed, separated into propositions, and analyzed according to the type of learning process they represented. Results demonstrated that diagrams promoted inference generation but did not affect other learning processes (such as elaboration or comprehension monitoring). However, only simple diagrams promoted generation of inferences that integrated domain information. Results indicate that diagrams may be useful because they guide the learner to engage in the cognitive processes required for deep understanding.

How does prior knowledge impact students' online learning behaviors?

This study explored the impact of prior domain knowledge on students’ strategies and use of digital resources during a Web-based learning task. Domain knowledge was measured using pre- and posttests of factual knowledge and knowledge application. Students utilized an age- and topic-relevant collection of 796 Web resources drawn from an existing educational digital library to revise essays that they had written prior to the online learning task. Following essay revision, participants self-reported their strategies for improving their essays. Screen-capture software was used to record all student interactions with Web-based resources and all modifications to their essays. Analyses examined the relationship between different levels of students’ prior knowledge and online learning behaviors, self-reported strategies, and learning outcomes. Findings demonstrated that higher levels of factual prior knowledge were associated with deeper learning and stronger use of digital resources, but that higher levels of deep prior knowledge were associated with less frequent use of online content and fewer deep revisions. These results suggest that factual knowledge can serve as a useful knowledge base during self-directed, online learning tasks, but deeper prior knowledge may lead novice learners to adopt suboptimal processes and behaviors.