Sofiane Medjkoune

HAMEX - A Handwritten and Audio Dataset of Mathematical Expressions

In this paper, we present HAMEX, a new public dataset that contains mathematical expressions available in their on-line handwritten form and in their audio spoken form. We have designed this dataset so that, given a mathematical expression, its handwritten signal and its audio signal can be used jointly to design multimodal recognition systems. Here, we describe the different steps that allowed us to acquire this dataset, from the creation of the mathematical expression corpora (including expressions from Wikipedia pages) to the segmentation and the transcription of the collected data, via the data collection process itself. Currently, the dataset contains 4 350 on-line handwritten mathematical expressions written by 58 writers, and the corresponding audio expressions (in French) spoken by 58 speakers. The ground truth is also provided both for the handwritten expressions (as INKML files with the digital ink, the symbol segmentation, and the MATHML structure) and for the audio expressions (as XML files with the transcriptions of the spoken expressions).

Mathematical Symbol Hypothesis Recognition with Rejection Option

In the context of handwritten mathematical expressions recognition, a first step consist on grouping strokes (segmentation) to form symbol hypotheses: groups of strokes that might represent a symbol. Then, the symbol recognition step needs to cope with the identification of wrong segmented symbols (false hypotheses). However, previous works on symbol recognition consider only correctly segmented symbols. In this work, we focus on the problem of mathematical symbol recognition where false hypotheses need to be identified. We extract symbol hypotheses from complete handwritten mathematical expressions and train artificial neural networks to perform both symbol classification of true hypotheses and rejection of false hypotheses. We propose a new shape context-based symbol descriptor: fuzzy shape context. Evaluation is performed on a publicly available dataset that contains 101 symbol classes. Results show that the fuzzy shape context version outperforms the original shape context. Best recognition and false acceptance rates were obtained using a combination of shape contexts and online features: 86% and 17.5% respectively. As false rejection rate, we obtained 8.6% using only online features.

Handwritten and Audio Information Fusion for Mathematical Symbol Recognition

Considerable efforts are being done within the scientific community to make as easier as possible the way that the human being converses with its machine. Handwriting and speech are two common ways used to achieve this goal and are probably among those which attracted much interest. In mathematical content recognition tasks, these two modalities are used with a certain success. This paper presents an architecture based on a speech handwriting data fusion for isolated mathematical symbol recognition. Different fusion methods are explored. The results are very encouraging since recognition rates are increased comparatively to mono modality approaches.

Text Alignment from Bimodal Mathematical Expression Sources

In this paper we propose a new approach to merge mathematical expression recognition results coming from handwriting and speech modalities. Using a bimodal description of mathematical expressions allows taking advantage of the complementarities between both signals, and can disambiguate situations were a single modality would not be clear enough. To combine the signals coming from both modalities, we propose to represent them in the same space as a textual description. First, from the handwriting signal, we generate the Nbest mathematical expressions, each of them is next translated as different possible strings. From the audio signal, an automatic speech recognition system provides a transcript, which is also available as a string. A string comparison algorithm is achieved to select the best mathematical expressions. This bimodal system is evaluated on real bimodal data from the HAMEX dataset and the results are compared to a single modality (handwriting) based system.

Multimodal mathematical expressions recognition: case of speech and handwriting

In this work, we propose to combine two modalities, handwriting and speech, to build a mathematical expression recognition system. Based on two sub-systems which process each modality, we explore various fusion methods to resolve ambiguities which naturally occur independently. The results that are reported on the HAMEX bimodal database show an improvement with respect to a mono-modal based system.The work reported in this paper concerns the problem of mathematical expressions recognition. This task is known to be a very hard one. We propose to alleviate the difficulties by taking into account two complementary modalities. The modalities referred to are handwriting and audio ones. To combine the signals coming from both modalities, various fusion methods are explored. Performances evaluated on the HAMEX dataset show a significant improvement compared to a single modality (handwriting) based system.

Using Speech for Handwritten Mathematical Expression Recognition Disambiguation

The main goal of this work is to set up a multimodal system dedicated to mathematical expression recognition. In the proposed architecture, the transcription coming out from a speech recognition system is used to disambiguate the result of a handwriting recognition module. A set of keywords is built from the transcription module and used to rescore the outputs of both the handwriting classifier and the structural analysis module. Performances evaluated on the HAMEX dataset show a significant improvement over a single modality system.

Combining Speech and Handwriting Modalities for Mathematical Expression Recognition

In this paper, we open new perspectives for mathematical expression recognition by introducing an original bimodal system. Since handwritten mathematical expression recognition is a very challenging task prone to many ambiguities, we use speech as an additional modality to circumvent limitations that are inherent to the written form. A use case scenario corresponds to lectures given in classrooms where the teacher would write and read aloud any mathematical expressions to allow a better interpretation. In addition to state-of-the-art solutions for recognizing handwriting and speech, we introduce a multilayer architecture for the merger of modalities. Specifically, the Dempster–Shafer theory is used to process the information at the symbol level. This bimodal system is evaluated on real bimodal data, the HAMEX dataset. Large improvements are observed when speech and handwriting are combined when compared to the single handwriting modality.

Approches multimodales pour la reconnaissance d'expressions mathématiques. Collaboration des modalités audio et écriture manuscrite en-ligne.

Dans cet article, nous presentons un systeme original de reconnaissance d’expressions mathematiques dans un cadre bimodal. En disposant a la fois du signal de la parole, correspondant a la dictee de l’expression, et du signal manuscrit, correspondant a son ecriture manuscrite en-ligne, nous proposons une architecture permettant de combiner ces deux modalites. Nous avons pu mesurer sur la base bimodale HAMEX, les performances respectives des systemes mono ou multimodalites, et quantifier la desambiguisation atteinte grâce a la complementarite des deux sources d’information.

Using online handwriting and audio streams for mathematical expressions recognition: a bimodal approach

The work reported in this paper concerns the problem of mathematical expressions recognition. This task is known to be a very hard one. We propose to alleviate the difficulties by taking into account two complementary modalities. The modalities referred to are handwriting and audio ones. To combine the signals coming from both modalities, various fusion methods are explored. Performances evaluated on the HAMEX dataset show a significant improvement compared to a single modality (handwriting) based system.

Using Online Handwriting and Audio Streams for Mathematical Expressions Recognition: a Bimodal Approach

In this work, we propose to combine two modalities, handwriting and speech, to build a mathematical expression recognition system. Based on two sub-systems which process each modality, we explore various fusion methods to resolve ambiguities which naturally occur independently. The results that are reported on the HAMEX bimodal database show an improvement with respect to a mono-modal based system.The work reported in this paper concerns the problem of mathematical expressions recognition. This task is known to be a very hard one. We propose to alleviate the difficulties by taking into account two complementary modalities. The modalities referred to are handwriting and audio ones. To combine the signals coming from both modalities, various fusion methods are explored. Performances evaluated on the HAMEX dataset show a significant improvement compared to a single modality (handwriting) based system.