James V. Mahoney

Interpreting Sloppy Stick Figures by Graph Rectification and Constraint-Based Matching

Programs for understanding hand-drawn sketches and diagrams must interpret curvilinear configurations that are sloppily drawn and highly variable in form. We propose a two-stage subgraph matching framework for sketch recognition that can accommodate great variability in form and yet provide efficient matching and easy extensibility to new configurations. First, a rectification stage corrects the initial data graph for the common deviations of each kind of constituent local configuration from its ideal form. The model graph is then matched directly to the data by a constraint-based subgraph matching process, without the need for complex error-tolerance. We explore the approach in the domain of human stick figures in diverse poses.

Perceptual Support of Diagram Creation and Editing

Diagrams mediate thinking and understanding largely through the human visual system’s innate ability to perceive visuo-spatial structure. Tools for working with diagrams will benefit from the ability of machines to identify visual structure in concert with their human users. This poster and its companion summarize recent progress in perceptually-supported diagram creation and editing. In particular, our research group has deployed a document image editing program realizing some measure of Gestalt Perceptual Organization for sketches and diagrams.

ScanScribe: Perceptually Supported Diagram Image Editing

We have implemented a prototype document image editor that incorporates principles and algorithms for Perceptual Organization in order to facilitate selection and manipulation of visually salient image objects. Called ScanScribe, the program serves two purposes. First, it offers an illustration of the advantageous use of Gestalt principles of segmentation and grouping of text and line-art found in diagrams. Second, it is a practical tool for modifying existing diagrams and composing new diagrams from mixed source material. ScanScribe’s user interface design and foundational representations are designed to scale to support recognition of domain objects found by structural matching through subgraph correspondence or other techniques.

Interpreting Sloppy Stick Figures with Constraint-Based Subgraph Matching

Machine systems for understanding hand-drawn sketches must reliably interpret common but sloppy curvilinear configurations. The task is commonly expressed as finding an image model in the image data, but few approaches exist for recognizing drawings with missing model parts and noisy data. In this paper, we propose a two-stage structural modeling approach that combines computer vision techniques with constraint-based recognition. The first stage produces a data graph through standard image analysis techniques augmented by rectification operations that account for common forms of drawing variability and noise. The second stage combines CLP(FD) with concurrent constraint programming for efficient and optimal matching of attributed model and data graphs. This approach offers considerable ease in stating model constraints and objectives, and also leads to an efficient algorithm that scales well with increasing image complexity.

Perceptual Organization as a Foundation for Graphics Recognition

This paper motivates an approach to graphics recognition grounded in a framework for human and machine vision known as Perceptual Organization. We review some of the characteristics of this approach that distinguish it from traditional engineering of document recognition systems, and we suggest why and how the techniques and philosophy of Perceptual Organization might lead to advances in the very practical matters of interpreting diagrams, drawings, and sketches.