Kenny Davila

Using Off-Line Features and Synthetic Data for On-Line Handwritten Math Symbol Recognition.

We present an approach for on-line recognition of handwritten math symbols using adaptations of off-line features and synthetic data generation. We compare the performance of our approach using four different classification methods: AdaBoost. M1 with C4.5 decision trees, Random Forests and Support-Vector Machines with linear and Gaussian kernels. Despite the fact that timing information can be extracted from on-line data, our feature set is based on shape description for greater tolerance to variations of the drawing process. Our main datasets come from the Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME) 2012 and 2013. Class representation bias in CROHME datasets is mitigated by generating samples for underrepresented classes using an elastic distortion model. Our results show that generation of synthetic data for underrepresented classes might lead to improvements of the average per-class accuracy. We also tested our system using the Math Brush dataset achieving a top-1 accuracy of 89.87% which is comparable with the best results of other recently published approaches on the same dataset.

Multi-Stage Math Formula Search: Using Appearance-Based Similarity Metrics at Scale

When using a mathematical formula for search (query-by-expression), the suitability of retrieved formulae often depends more upon symbol identities and layout than deep mathematical semantics. Using a Symbol Layout Tree representation for formula appearance, we propose the Maximum Subtree Similarity (MSS) for ranking formulae based upon the subexpression whose symbols and layout best match a query formula. Because MSS is too expensive to apply against a complete collection, the Tangent-3 system first retrieves expressions using an inverted index over symbol pair relationships, ranking hits using the Dice coefficient; the top-k formulae are then re-ranked by MSS. Tangent-3 obtains state-of-the-art performance on the NTCIR-11 Wikipedia formula retrieval benchmark, and is efficient in terms of both space and time. Retrieval systems for other graphical forms, including chemical diagrams, flowcharts, figures, and tables, may benefit from adopting this approach.

Shape matching using keypoints extracted from both the foreground and the background of binary images

We introduce a descriptor for shape feature extraction and matching using keypoints that are extracted from both the foreground and the background of binary images. First, distance transform (DT) is applied on the image after contour detection. Then, connected components (CCs) of pixels having the same intensity are extracted. Keypoints correspond to centers of mass of CCs. A keypoint filtering mechanism is applied by estimating the spatial stability of keypoints when successive iterations of image blurring and binarization are applied. Finally, features are extracted for each keypoint using a round layout which radius is set depending on the keypoints location. We evaluate our descriptor using datasets of silhouette images, handwritten math expressions, and logos. Experimental results show that our descriptor is competitive compared with state-of-the-art methods, and that keypoint filtering is effective in reducing the number of keypoints without compromising matching performances.

Appearance-Based Retrieval of Mathematical Notation in Documents and Lecture Videos

Large data collections containing millions of math formulae in different formats are available on-line. Retrieving math expressions from these collections is challenging. Based on the notion that visually similar formulas are related, we propose a framework for appearance-based formula retrieval in two different modalities: symbolic for text documents and image-Based for videos. We believe that we can achieve high quality formula retrieval results using the visual appearance of math notation without complex formula semantic analysis. We represent mathematical notation using different graph types to take advantage of the information available on each domain. For symbolic formula retrieval, math expressions in text formats like LaTeX are parsed to generate Symbol Layout Trees. For image-based formula retrieval, image processing techniques are used to create a graph-based image content representation. We store these graphs using an inverted index of pairs of primitives defined by the triplet (p, q, r), where p and q are the labels of two primitives connected in the graph by the path r. Retrieval is a two-stage process: candidate selection and reranking. The first stage uses pairs of primitives from the query graph to find matches in the inverted index. Each match is given an initial score using the Dice coefficient of matched pairs of primitives. The best top-K candidates from the first stage are selected for re-ranking using a detailed similarity metric. Two steps are performed for each candidate: matching and scoring. The matching step is done by searching for the largest common substructure between query and candidate graphs. Matching is related to the problem of finding the maximum common subgraph isomorphism (MCS) between two graphs. In addition, we consider label unification for symbolic formula retrieval, and our wildcard query nodes can match entire subgraphs. In the scoring step, multiple similarity criteria define a score vector used to sort candidates, either by lexicographic order or by a function of these scores. Different datasets and benchmarks will be required to evaluate each modality. For symbolic formula retrieval, we will use the most recent versions of the NTCIR MathIR Tasks benchmarks. To the best of our knowledge, there are no benchmarks for large scale image-based formula retrieval. However, the same collections used for symbolic formula retrieval could be adapted by rendering math expressions to images. In addition, we will use datasets of math lecture videos for image-based formula retrieval. Traditional graded-scales of relevance used for evaluation of retrieval systems have been shown to have inconsistency issues. We plan to use pairwise candidate comparisons during our evaluation phase. Some aggregation methods exist that generate relevance scores and ideal rankings using these pairwise candidate comparisons. The proposed framework can be adapted to work for other domains like chemistry or technical diagrams where visually similar elements are usually related.

Accessmath: Indexing and retrieving video segments containing math expressions based on visual similarity

Access Math project is a work in progress oriented toward helping visually impaired students in and out of the class-room. The system works with videos from math lectures. For each lecture, videos of the whiteboard content from two different sources are provided. An application for extraction and retrieval of that content is presented. After the content has been indexed, the user can select a portion of the whiteboard content found in a video frame and use it as a query to find segments of video with similar content. Graphs of neighboring connected components are used to describe both the query and the candidate regions, and the results of a query are ranked using the recall of matched graph edges between the graph of the query and the graph of each candidate. This is a recognition-free method and belongs to the field of sketch-based image retrieval.

Layout and Semantics: Combining Representations for Mathematical Formula Search

Math-aware search engines need to support formulae in queries. Mathematical expressions are typically represented as trees defining their operational semantics or visual layout. We propose searching both formula representations using a three-layer model. The first layer selects candidates using spectral matching over tree node pairs. The second layer aligns a query with candidates and computes similarity scores based on structural matching. In the third layer, similarity scores are combined using linear regression. The two representations are combined using retrieval in parallel indices and regression over similarity scores. For NTCIR-12 Wikipedia Formula Browsing task relevance rankings, we see each layer increasing ranking quality and improved results when combining representations as measured by Bpref and nDCG scores.