Jeffrey M. Usher

CogSketch: Sketch Understanding for Cognitive Science Research and for Education

Sketching is a powerful means of working out and communicating ideas. Sketch understanding involves a combination of visual, spatial, and conceptual knowledge and reasoning, which makes it both challenging to model and potentially illuminating for cognitive science. This paper describes CogSketch, an ongoing effort of the NSF-funded Spatial Intelligence and Learning Center, which is being developed both as a research instrument for cognitive science and as a platform for sketch-based educational software. We describe the idea of open-domain sketch understanding, the scientific hypotheses underlying CogSketch, and provide an overview of the models it employs, illustrated by simulation studies and ongoing experiments in creating sketch-based educational software.

Sketching for knowledge capture: a progress report

Many concepts and situations are best explained by sketching. This paper describes our work on sKEA, the sketching Knowledge Entry Associate, a system designed for knowledge capture via sketching. We discuss the key ideas of sKEA: blob semantics for glyphs to sidestep recognition for visual symbols, qualitative spatial reasoning to provide richer visual and conceptual understanding of what is being communicated, arrows to express domain relationships, layers to express within-sketch segmentation (including a meta-layer to express subsketch relationships themselves via sketching), and analogical comparison to explore similarities and differences between sketched concepts. Experiences with sKEA to date and future plans are also discussed.

CogSketch: open-domain sketch understanding for cognitive science research and for education

In this paper, we describe CogSketch, an open-domain sketch understanding system built on the nuSketch architecture. CogSketch captures the multi-modal, unconstrained nature of sketching by focusing on reasoning over recognition. We describe this approach, as well as two application domains for CogSketch: cognitive modeling, and education.

Qualitative spatial reasoning about sketch maps

Sketch maps are an important spatial representation used in many geospatial-reasoning tasks. This article describes techniques we have developed that enable software to perform humanlike reasoning about sketch maps. We illustrate the utility of these techniques in the context of nuSketch Battlespace, a research system that has been successfully used in a variety of experiments. After an overview of the nuSketch approach and nuSketch Battlespace, we outline the representations of glyphs and sketches and the nuSketch spatial reasoning architecture. We describe the use of qualitative topology and Voronoi diagrams to construct spatial representations, and explain how these facilities are combined with analogical reasoning to provide a simple form of enemy intent hypothesis generation.

Sketching for military courses of action diagrams

A serious barrier to the digitalization of the US military is that commanders find traditional mouse/menu, CAD-style interfaces unnatural. Military commanders develop and communicate battle plans by sketching courses of action (COAs). This paper describes nuSketch Battlespace, the latest version in an evolving line of sketching interfaces that commanders find natural, yet supports significant increased automation. We describe techniques that should be applicable to any specialized sketching domain: glyph bars and compositional symbols to tractably handle the large number of entities that military domains use, specialized glyph types and gestures to keep drawing tractable and natural, qualitative spatial reasoning to provide sketch-based visual reasoning, and comic graphs to describe multiple states and plans. Experiments, both completed and in progress, are described to provide evidence as to the utility of the system.

SketchWorksheets: A Brief Summary

Sketch worksheets are a new kind of sketch-based education software designed to facilitate spatial learning. Each worksheet represents a particular exercise, which the student does on a computer. Students get feedback, based on automatic comparison of their sketch with a hidden solution sketch. A software gradebook, which uses scoring rubrics in the solution sketch, is intended to help instructors in grading. Sketch worksheets have been used in classroom experiments with college students, high school students, and middle-school students. They are domain-independent, requiring only that the exercise involves visual distinctions that the software can understand. They rely on cognitive models of human visual/spatial representations and a model of human analogical matching. An authoring environment enables domain instructors to create new sketch worksheets on their own. CogSketch is freely available for download.

Sketching for knowledge capture: a demonstration

Many concepts and situations are best explained by sketching. This demonstration will show the key ideas underlying sKEA, the sketching knowledge entry associate, a system we have built for knowledge capture via sketching. In particular, we will demonstrateHow glyph bars and blob semantics are used to sidestep the need for recognition of visual symbols. The use of qualitative spatial reasoning to provide richer visual and conceptual understanding of what is being communicatedHow arrows are used to express domain relationshipsThe use of layers to express within-sketch segmentation, including a meta-layer to express subsketch relationships themselves via sketchingUsing analogical comparison to explore similarities and differences between sketched concepts.