Scott MacLean

MathBrush: A System for Doing Math on Pen-Based Devices

Many on-line (interactive) mathematics recognition systems allow the creation of typeset equations, normally in LaTeX, but they do not support mathematical problem solving. In this paper, we present MathBrush, a system that allows users to draw math input using a pen-input device on a tablet computer, recognizes the math expression, and then supports mathematical transformation and problem solving using back-end Computer Algebra Systems (CAS). We describe the architecture of the MathBrush system, which includes modules that support symbol recognition, semantic analysis, the transfer of recognized expressions to back-end CAS, and interface techniques for interacting with CAS output. We also identify unique challenges associated with recognition for math problem solving, such as the need for deeper semantic analysis than is required by LATEX, and the need to deal with ambiguities in user input. Our experiences serve to inform researchers seeking to design interactive mathematics recognition systems geared toward mathematical problem solving.

A new approach for recognizing handwritten mathematics using relational grammars and fuzzy sets

We present a new approach for parsing two-dimensional input using relational grammars and fuzzy sets. A fast, incremental parsing algorithm is developed, motivated by the two-dimensional structure of written mathematics. The approach reports all identifiable parses of the input. The parses are represented as a fuzzy set, in which the membership grade of a parse measures the similarity between it and the handwritten input. To identify and report parses efficiently, we adapt and apply existing techniques such as rectangular partitions and shared parse forests, and introduce new ideas such as relational classes and interchangeability. We also present a correction mechanism that allows users to navigate parse results and choose the correct interpretation in case of recognition errors or ambiguity. Such corrections are incorporated into subsequent incremental recognition results. Finally, we include two empirical evaluations of our recognizer. One uses a novel user-oriented correction count metric, while the other replicates the CROHME 2011 math recognition contest. Both evaluations demonstrate the effectiveness of our proposed approach.

Grammar-based techniques for creating ground-truthed sketch corpora

Although publicly available, ground-truthed corpora have proven useful for training, evaluating, and comparing recognition systems in many domains, the availability of such corpora for sketch recognizers, and math recognizers in particular, is currently quite poor. This paper presents a general approach to creating large, ground-truthed corpora for structured sketch domains such as mathematics. In the approach, random sketch templates are generated automatically using a grammar model of the sketch domain. These templates are transcribed manually, then automatically annotated with ground-truth. The annotation procedure uses the generated sketch templates to find a matching between transcribed and generated symbols. A large, ground-truthed corpus of handwritten mathematical expressions presented in the paper illustrates the utility of the approach.

Is the iPad useful for sketch input?: a comparison with the tablet PC

Despite the increasing prevalence of touch-based tablet devices, little attention has been paid to what effects, if any, this form factor has on sketch behaviours in general and on sketch recognizers in particular. We investigate the title question through an empirical study in the context of mathematical expression recognition. Using a corpus of thirty expressions drawn on Tablet PC and iPad by thirty writers, we show that characteristics of sketching and drawing differ depending on platform. While recognition is most accurate on the Tablet PC due to its technical superiority, recognition is feasible on mobile touch-based devices. However, there are characteristics of sketching on multi-touch tablets that differ, and these physical characteristics of writing impact recognition accuracy. Together, our observations inform the broader Sketch Recognition community as they design systems targeted to multi-touch tablets.

A Bayesian model for recognizing handwritten mathematical expressions

Recognizing handwritten mathematics is a challenging classification problem, requiring simultaneous identification of all the symbols comprising an input as well as the complex two-dimensional relationships between symbols and subexpressions. Because of the ambiguity present in handwritten input, it is often unrealistic to hope for consistently perfect recognition accuracy. We present a system which captures all recognizable interpretations of the input and organizes them in a parse forest from which individual parse trees may be extracted and reported. If the top-ranked interpretation is incorrect, the user may request alternates and select the recognition result they desire. The tree extraction step uses a novel probabilistic tree scoring strategy in which a Bayesian network is constructed based on the structure of the input, and each joint variable assignment corresponds to a different parse tree. Parse trees are then reported in order of decreasing probability. Two accuracy evaluations demonstrate that the resulting recognition system is more accurate than previous versions (which used non-probabilistic methods) and other academic math recognizers. HighlightsA recognizer for hand-drawn mathematical expressions.Relational context-free grammars structure hand-drawn mathematical input.Identifies multiple interpretations of the input based on output of lower-level classifiers.Bayesian probabilistic model for scoring and comparing recognized parse trees.Accuracy exceeds that of other academic recognizers.

MathBrush: a case study for pen-based interactive mathematics

Current generations of computer algebra systems require users to transform two dimensional math expressions into one dimensional strings, to master complex sets of commands, and to analyze lengthy output strings for relevant information. MathBrush is a system, designed based on research in education pedagogy, that provides a pen-based interface to many of the features of computer algebra systems. We describe relevant work in education pedagogy as a motivation for MathBrushs design. We highlight aspects of MathBrush that are unique from other contemporary pen-math systems. Finally, we present the results of a thinkaloud evaluation of the MathBrush system. Together, these observations validate aspects of the current design of MathBrush, suggest areas for refinement, and inform the design of future pen-math systems.

Tools for the efficient generation of hand-drawn corpora based on context-free grammars

In sketch recognition systems, ground-truth data sets serve to both train and test recognition algorithms. Unfortunately, generating data sets that are sufficiently large and varied is frequently a costly and time-consuming endeavour. In this paper, we present a novel technique for creating a large and varied ground-truthed corpus for hand drawn math recognition. Candidate math expressions for the corpus are generated via random walks through a context-free grammar, the expressions are transcribed by human writers, and an algorithm automatically generates ground-truth data for individual symbols and inter-symbol relationships within the math expressions. While the techniques we develop in this paper are illustrated through the creation of a ground-truthed corpus of mathematical expressions, they are applicable to any sketching domain that can be described by a formal grammar.