Mirette S. Marzouk

MultiView: Multilevel video content representation and retrieval

In this article, several practical algorithms are proposed to support content-based video analysis, modeling, representation, summarization, indexing, and access. First, a multilevel video data- base model is given. One advantage of this model is that it provides a reasonable approach to bridging the gap between low-level rep- resentative features and high-level semantic concepts from a hu- man point of view. Second, several model-based video analysis techniques are proposed. In order to detect the video shots, we present a novel technique, which can adapt the threshold for scene cut detection to the activities of variant videos or even different video shots. A seeded region aggregation and temporal tracking technique is proposed for generating the semantic video objects. The semantic video scenes can then be generated from these extracted video access units (e.g., shots and objects) according to some domain knowledge. Third, in order to categorize video contents into a set of semantic clusters, an integrated video classification technique is de- veloped to support more efficient multilevel video representation, summarization, indexing, and access techniques.

MathBrush: A System for Doing Math on Pen-Based Devices

Many on-line (interactive) mathematics recognition systems allow the creation of typeset equations, normally in LaTeX, but they do not support mathematical problem solving. In this paper, we present MathBrush, a system that allows users to draw math input using a pen-input device on a tablet computer, recognizes the math expression, and then supports mathematical transformation and problem solving using back-end Computer Algebra Systems (CAS). We describe the architecture of the MathBrush system, which includes modules that support symbol recognition, semantic analysis, the transfer of recognized expressions to back-end CAS, and interface techniques for interacting with CAS output. We also identify unique challenges associated with recognition for math problem solving, such as the need for deeper semantic analysis than is required by LATEX, and the need to deal with ambiguities in user input. Our experiences serve to inform researchers seeking to design interactive mathematics recognition systems geared toward mathematical problem solving.

Grammar-based techniques for creating ground-truthed sketch corpora

Although publicly available, ground-truthed corpora have proven useful for training, evaluating, and comparing recognition systems in many domains, the availability of such corpora for sketch recognizers, and math recognizers in particular, is currently quite poor. This paper presents a general approach to creating large, ground-truthed corpora for structured sketch domains such as mathematics. In the approach, random sketch templates are generated automatically using a grammar model of the sketch domain. These templates are transcribed manually, then automatically annotated with ground-truth. The annotation procedure uses the generated sketch templates to find a matching between transcribed and generated symbols. A large, ground-truthed corpus of handwritten mathematical expressions presented in the paper illustrates the utility of the approach.

Is the iPad useful for sketch input?: a comparison with the tablet PC

Despite the increasing prevalence of touch-based tablet devices, little attention has been paid to what effects, if any, this form factor has on sketch behaviours in general and on sketch recognizers in particular. We investigate the title question through an empirical study in the context of mathematical expression recognition. Using a corpus of thirty expressions drawn on Tablet PC and iPad by thirty writers, we show that characteristics of sketching and drawing differ depending on platform. While recognition is most accurate on the Tablet PC due to its technical superiority, recognition is feasible on mobile touch-based devices. However, there are characteristics of sketching on multi-touch tablets that differ, and these physical characteristics of writing impact recognition accuracy. Together, our observations inform the broader Sketch Recognition community as they design systems targeted to multi-touch tablets.

Video query processing in the VDBMS testbed for video database research

The increased use of video data sets for multimedia-based applications has created a demand for strong video database support, including efficient methods for handling the content-based query and retrieval of video data. Video query processing presents significant research challenges, mainly associated with the size, complexity and unstructured nature of video data. A video query processor must support video operations for search by content and streaming, new query types, and the incorporation of video methods and operators in generating, optimizing and executing query plans. In this paper, we address these query processing issues in two contexts, first as applied to the video data type and then as applied to the stream data type. We first present the query processing functionality of the VDBMS video database management system as a framework designed to support the full range of functionality for video as an abstract data type. We describe two query operators for the video data type which implement the rank-join and stop-after algorithms. As videos may be considered streams of consecutive image frames, video query processing can be expressed as continuous queries over video data streams. The stream data type was therefore introduced into the VDBMS system, and system functionality was extended to support general data streams. From this viewpoint, we present an approach for defining and processing streams, including video, through the query execution engine. We describe the implementation of several algorithms for video query processing expressed as continuous queries over video streams, such as fast forward, region-based blurring and left outer join. We include a description of the window-join algorithm as a core operator for continuous query systems, and discuss shared execution as an optimization approach for stream query processing.

MathBrush: a case study for pen-based interactive mathematics

Current generations of computer algebra systems require users to transform two dimensional math expressions into one dimensional strings, to master complex sets of commands, and to analyze lengthy output strings for relevant information. MathBrush is a system, designed based on research in education pedagogy, that provides a pen-based interface to many of the features of computer algebra systems. We describe relevant work in education pedagogy as a motivation for MathBrushs design. We highlight aspects of MathBrush that are unique from other contemporary pen-math systems. Finally, we present the results of a thinkaloud evaluation of the MathBrush system. Together, these observations validate aspects of the current design of MathBrush, suggest areas for refinement, and inform the design of future pen-math systems.

Tools for the efficient generation of hand-drawn corpora based on context-free grammars

In sketch recognition systems, ground-truth data sets serve to both train and test recognition algorithms. Unfortunately, generating data sets that are sufficiently large and varied is frequently a costly and time-consuming endeavour. In this paper, we present a novel technique for creating a large and varied ground-truthed corpus for hand drawn math recognition. Candidate math expressions for the corpus are generated via random walks through a context-free grammar, the expressions are transcribed by human writers, and an algorithm automatically generates ground-truth data for individual symbols and inter-symbol relationships within the math expressions. While the techniques we develop in this paper are illustrated through the creation of a ground-truthed corpus of mathematical expressions, they are applicable to any sketching domain that can be described by a formal grammar.

VDBMS: A testbed facility for research in video database benchmarking

Abstract.Real-world video-based applications require database technology that is capable of storing digital video in the form of video databases and providing content-based video search and retrieval. Methods for handling traditional data storage, query, search, retrieval, and presentation cannot be extended to provide this functionality. The VDBMS research initiative is motivated by the requirements of video-based applications to search and retrieve portions of video data based on content and by the need for testbed facilities to facilitate research in the area of video database management. In this paper we describe the VDBMS video database research platform, a system that supports comprehensive and efficient database management for digital video. Our fundamental concept is to provide a full range of functionality for video as a well-defined abstract database data type, with its own description, parameters, and applicable methods. Research problems that are addressed by VDBMS to support the handling of video data include MPEG7 standard multimedia content representation, algorithms for image-based shot detection, image processing techniques for extracting low-level visual features, a high-dimensional indexing technique to access the high-dimensional feature vectors extracted by image preprocessing, multimedia query processing and optimization, new query operators, real-time stream management, a search-based buffer management policy, and an access control model for selective, content-based access to streaming video. VDBMS also provides an environment for testing the correctness and scope of new video processing techniques, measuring the performance of algorithms in a standardized way, and comparing the performance of different implementations of an algorithm or component. We are currently developing video component wrappers with well-defined interfaces to facilitate the modification or replacement of video processing components. The ultimate goal of the VDBMS project is a flexible, extensible framework that can be used by the research community for developing, testing, and benchmarking video database technologies.

A distributed database server for continuous media

In our project, we are adopting a new approach for handling video data. We view the video as a well-defined data type with its own description, parameters and applicable methods. The system is based on PREDATOR, an open-source object-relational DBMS. PREDATOR uses Shore as the underlying storage manager. Supporting video operations (storing, searching-by-content and streaming) and new query types (query-by-example and multi-feature similarity searching) requires major changes in many of the traditional system components. More specifically, the storage and buffer manager has to deal with huge volumes of data with real-time constraints. Query processing has to consider the video methods and operators in generating, optimizing and executing the query plans.

Managing ambiguity in mathematical matrices

In this paper we describe strategies for recognizing and using hand drawn matrices in a pen math system. This includes a new technique to recognize common short-forms of writing matrices using ellipsis (...). Ellipsis are commonly used in sketched matrices to illustrate the structure of a matrix without fully specifying the matrix. A second contribution of this paper is a new method to estimate the parameters of the hand drawn matrix, such as the number and position of the rows and columns. This is done using a modified clustering algorithm, allowing one to reduce the number of hard-coded constraints.