Tracy Hammond

LADDER, a sketching language for user interface developers

Sketch recognition systems are currently being developed for many domains, but can be time consuming to build if they are to handle the intricacies of each domain. In order to aid sketch-based user interface developers, we have developed tools to simplify the development of a new sketch recognition interface. We created LADDER, a language to describe how sketched diagrams in a domain are drawn, displayed, and edited. We then automatically transform LADDER structural descriptions into domain specific shape recognizers, editing recognizers, and shape exhibitors for use in conjunction with a domain independent sketch recognition system, creating a sketch recognition system for that domain. We have tested our framework by writing several domain descriptions and automatically generating a domain specific sketch recognition system from each description.

Tahuti: a geometrical sketch recognition system for UML class diagrams

We have created and tested Tahuti, a dual-view sketch recognition environment for class diagrams in UML. The system is based on a multi-layer recognition framework which recognizes multi-stroke objects by their geometrical properties allowing users the freedom to draw naturally as they would on paper rather than requiring the user to draw the objects in a pre-defined manner. Users can draw and edit while viewing either their original strokes or the interpreted version of their strokes engendering user-autonomy in sketching. The experiments showed that users preferred Tahuti to a paint program and to Rational Rose™ because it combined the ease of drawing found in a paint program with the ease of editing available in a UML editor.

iCanDraw: using sketch recognition and corrective feedback to assist a user in drawing human faces

When asked to draw, many people are hesitant because they consider themselves unable to draw well. This paper describes the first system for a computer to provide direction and feedback for assisting a user to draw a human face as accurately as possible from an image. Face recognition is first used to model the features of a human face in an image, which the user wishes to replicate. Novel sketch recognition algorithms were developed to use the information provided by the face recognition to evaluate the hand-drawn face. Two design iterations and user studies led to nine design principles for providing such instruction, presenting reference media, giving corrective feedback, and receiving actions from the user. The result is a proof-of-concept application that can guide a person through step-by-step instruction and generated feedback toward producing his/her own sketch of a human face in a reference image.

Recognizing sketched multistroke primitives

Sketch recognition attempts to interpret the hand-sketched markings made by users on an electronic medium. Through recognition, sketches and diagrams can be interpreted and sent to simulators or other meaningful analyzers. Primitives are the basic building block shapes used by high-level visual grammars to describe the symbols of a given sketch domain. However, one limitation of these primitive recognizers is that they often only support basic shapes drawn with a single stroke. Furthermore, recognizers that do support multistroke primitives place additional constraints on users, such as temporal timeouts or modal button presses to signal shape completion. The goal of this research is twofold. First, we wanted to determine the drawing habits of most users. Our studies found multistroke primitives to be more prevalent than multiple primitives drawn in a single stroke. Additionally, our studies confirmed that threading is less frequent when there are more sides to a figure. Next, we developed an algorithm that is capable of recognizing multistroke primitives without requiring special drawing constraints. The algorithm uses a graph-building and search technique that takes advantage of Tarjans linear search algorithm, along with principles to determine the goodness of a fit. Our novel, constraint-free recognizer achieves accuracy rates of 96% on freely-drawn primitives.

Sketch recognition algorithms for comparing complex and unpredictable shapes

In an introductory Engineering course with an annual enrollment of over 1000 students, a professor has little option but to rely on multiple choice exams for midterms and finals. Furthermore, the teaching assistants are too overloaded to give detailed feedback on submitted homework assignments. We introduce Mechanix, a computer-assisted tutoring system for engineering students. Mechanix uses recognition of freehand sketches to provide instant, detailed, and formative feedback as the student progresses through each homework assignment, quiz, or exam. Free sketch recognition techniques allow students to solve free-body diagram and static truss problems as if they were using a pen and paper. The same recognition algorithms enable professors to add new unique problems simply by sketching out the correct answer. Mechanix is able to ease the burden of grading so that instructors can assign more free response questions, which provide a better measure of student progress than multiple choice questions do.

Object interaction detection using hand posture cues in an office setting

Activity recognition plays a key role in providing information for context-aware applications. When attempting to model activities, some researchers have looked towards Activity Theory, which theorizes that activities have objectives and are accomplished through interactions with tools and objects. The goal of this paper is to determine if hand posture can be used as a cue to determine the types of interactions a user has with objects in a desk/office environment. Furthermore, we wish to determine if hand posture is user-independent across all users when interacting with the same objects in a natural manner. Our experiments indicate that (a) hand posture can be used to determine object interaction, with accuracy rates around 97%, and (b) hand posture is dependent upon the individual user when users are allowed to interact with objects as they would naturally.

Free-sketch recognition: putting the chi in sketching

Sketch recognition techniques have generally fallen into two camps. Gesture-based techniques, such as those used by the Palm Pilots Graffiti, can provide high-accuracy, but require the user to learn a particular drawing style in order for shapes to be recognized. Free-sketch recognition allows users to draw shapes as they would naturally, but most current techniques have low accuracies or require significant domain-level tweaking to make them usable. Our goal is to recognize free-hand sketches with high accuracy by developing generalized techniques that work for a variety of domains, including design and education. This is a work-in-progress, but we have made significant advancements toward our over-arching goal.

Interactive learning of structural shape descriptions from automatically generated near-miss examples

Sketch interfaces provide more natural interaction than the traditional mouse and palette tool, but can be time consuming to build if they have to be built anew for each new domain. A shape description language, such as the LADDER language we created, can significantly reduce the time necessary to create a sketch interface by enabling automatic generation of the interface from a domain description. However, structural shape descriptions, whether written by users or created automatically by the computer, are frequently over- or under- constrained. We present a technique to debug over- and under-constrained shapes using a novel form of active learning that generates its own suspected near-miss examples. Using this technique we implemented a graphical debugging tool for use by sketch interface developers.

Sketch-based educational games: "drawing" kids away from traditional interfaces

Computer-based games and technologies can be significant aids for helping children learn. However, most computer-based games simply address the learning styles of visual and auditory learners. Sketch-based interfaces, however, can also address the needs of those children who learn better through tactile and kinesthetic approaches. Furthermore, sketch recognition can allow for automatic feedback to aid children without the explicit need for teacher to be present. In this paper, we present various sketch-based tools and games that promote tactile learning and entertainment for children.

World of workout: a contextual mobile RPG to encourage long term fitness

In todays digital world many individuals spend their day in front of a computer or mobile phone for entertainment. Individuals enjoy a more sedentary lifestyle from advances in technology. This is one of the leading factors contributing to a decrease in fitness level for large parts of the populations in developed countries. We want to design a mobile role-playing game (RPG) where the character evolves based on the exercises the user performs in reality. This design can motivate and persuade a potentially large demographic of users to engage in physical activity for an extended period of time through the enjoyment of an engaging game. This novel application has shown the capability of automatically identifying and counting the exercises performed by the user. This automatic activity recognition and numeration is performed solely through the accelerometer of a single smartphone held by the user while exercising. The type and amount of exercise improve the characters speed, strength, and stamina based on the type and amount of exercise performed.