James Herold

Technical Section: SpeedSeg: A technique for segmenting pen strokes using pen speed

We present SpeedSeg, a technique for segmenting pen strokes into lines and arcs. The technique uses pen speed information to help infer the segmentation intended by the drawer. To begin, an initial set of candidate segment points is identified. This set includes speed minima below a threshold, and curvature maxima at which the pen speed is also below a threshold. The ink between each pair of consecutive segment points is then classified as either a line or an arc, depending on which fits best. Next, a feedback process is employed, and segments are judiciously merged and split as necessary to improve the quality of the segmentation. In user studies, SpeedSeg performed accurately for new users. The studies also demonstrated that SpeedSegs accuracy is surprisingly insensitive to the values of many of the empirical parameters used by the technique. However, it is still possible to quickly tune the system to optimize performance for a given user. Finally, SpeedSeg outperformed a state-of-the-art segmentation algorithm.

The 1 ¢ Recognizer: a fast, accurate, and easy-to-implement handwritten gesture recognition technique

We present the One Cent Recognizer, an easy-to-implement, efficient, and accurate handwritten gesture recognizer. By applying time series recognition techniques, we have developed a minimally complex technique that is both much faster than and at least as accurate as the Dollar Recognizer. Additionally, the One Cent Recognizer is much easier to implement than the Dollar Recognizer. Our technique is primarily enabled by a simple and novel one-dimensional representation of handwritten pen strokes. This representation is intrinsically rotation invariant, allowing our technique to avoid costly rotate-and-check searches typically employed in prior template-based gesture recognition techniques. In experiments, our technique has proven to be two orders of magnitude faster than the Dollar Recognizer.

Technical Section: A machine learning approach to automatic stroke segmentation

We present ClassySeg, a technique for segmenting hand-drawn pen strokes into lines and arcs. ClassySeg employs machine learning techniques to infer the segmentation intended by the drawer. The technique begins by identifying a set of candidate segment windows, each comprising a curvature maximum and its neighboring points. Features are computed for each point in each window based on curvature and other geometric properties. Most of these features are adapted from numerous prior segmentation approaches, effectively combining their strengths. These features are used to train a statistical classifier to identify which candidate windows contain true segment points. ClassySeg is more accurate than previous techniques for both user-independent and user-optimized training conditions. More importantly, ClassySeg represents a movement away from prior, heuristic-based approaches, toward a more general and extensible technique.

Newtons Pen II: an intelligent, sketch-based tutoring system and its sketch processing techniques

We present a pen-based intelligent tutoring system (ITS) for undergraduate Statics which scaffolds students in the construction of free body diagrams and equilibrium equations. Most existing ITSs rely on traditional WIMP (Windows, Icons, Menus, Pointers) interfaces, which often require the student to select the correct answer from among a set of predefined choices. Our system, by contrast, guides students in constructing solutions from scratch, mirroring the way they ordinarily solve problems, which recent research suggests is important for effective instruction. Our system employs several new techniques for sketch understanding, including a simple-to-implement stroke merging technique, a stroke clustering technique, and a technique that uses a Hidden Markov Model to correct interpretation errors in equations. Our tutoring system was deployed in an undergraduate Statics course at our university. Attitudinal surveys indicate that the tutoring system is preferable to traditional WIMP-based systems and is an effective educational tool.

How to make a Quick

We present Quick

: using hierarchical clustering to improve the efficiency of the Dollar Recognizer

(QuickBuck), an extension to the Dollar Recognizer designed to improve recognition efficiency. While the Dollar Recognizer must search all training templates to recognize an unknown symbol, Quick

Using speech to identify gesture pen strokes in collaborative, multimodal device descriptions

employs hierarchical clustering along with branch and bound search to do this more efficiently. Experiments have demonstrated that Quick

ClassySeg: a machine learning approach to automatic stroke segmentation

is almost always faster than the Dollar Recognizer and always selects the same best-match templates.

Characterizing Students Handwritten Self-explanations

Abstract One challenge in building collaborative design tools that use speech and sketch input is distinguishing gesture pen strokes from those representing device structure, that is, object strokes. In previous work, we developed a gesture/object classifier that uses features computed from the pen strokes and the speech aligned with them. Experiments indicated that the speech features were the most important for distinguishing gestures, thus indicating the critical importance of the speech–sketch alignment. Consequently, we have developed a new alignment technique that employs a two-step process: the speech is first explicitly segmented (primarily into clauses), then the segments are aligned with the pen strokes. Our speech segmentation step is unique in that it uses sketch features for locating segment boundaries in multimodal dialog. In addition, it uses a single classifier to directly combine word-based, prosodic (pause), and sketch-based features. In the second step, segments are initially aligned with strokes based on temporal correlation, and then classifiers are used to detect and correct two common alignment errors. Our two-step technique has proven to be substantially more accurate at alignment than the existing technique that lacked explicit segmentation. It is more important that, for nearly all cases, our new technique results in greater gesture classification accuracy than the existing technique, and performed nearly as well as the benchmark manual speech–sketch alignment.

Automatic Handwritten Statics Solution Classification and its Applications in Predicting Student Performance

We present ClassySeg, a technique for segmenting hand-drawn pen strokes into lines and arcs. ClassySeg employs machine learning techniques to infer the segmentation intended by the drawer. The technique begins by identifying a set of candidate segment points, consisting of all curvature maxima. Features are computed for each candidate point based on speed, curvature, and other geometric properties. These features are adapted from numerous prior segmentation approaches, effectively combining their strengths. These features are used to train a statistical classifier to identify which candidate points are true segment points. A beam search is used to approximate the optimal subset of features to use as input to the classifier. ClassySeg is more accurate than previous techniques for user-independent training conditions, and is as good as the current state-of-the-art algorithm for user-optimized conditions. More importantly, ClassySeg represents a movement away from prior heuristic-based approaches towards a more general and extensible approach.