Xugang Wang

Structuralizing digital ink for efficient selection

Raw digital ink is informal and unstructured. Its editing, especially its selection, is often inefficient. In this paper, we present approaches to structuralize raw digital ink as multiple hierarchies to facilitate its selection. First a link model is built to organize ink as a mesh-like structure. Based on the link model, the isolated stroke groups form patches. In each patch, textual and graphical areas are separated. Then, each textual area is segmented into text lines, and each text line is partitioned to words. We also design gestures for selecting structured ink. Experiments showed that our ink-structuralizing approaches are effective and selecting structured ink by our gestures considerably outperforms selecting raw ink.

Intelligent understanding of handwritten geometry theorem proving

Computer-based geometry systems have been widely used for teaching and learning, but largely based on mouse-and-keyboard interaction, these systems usually require users to draw figures by following strict task structures defined by menus, buttons, and mouse and keyboard actions. Pen-based designs offer a more natural way to develop geometry theorem proofs with hand-drawn figures and scripts. This paper describes a pen-based geometry theorem proving system that can effectively recognize hand-drawn figures and hand-written proof scripts, and accurately establish the correspondence between geometric components and proof steps. Our system provides dynamic and intelligent visual assistance to help users understand the process of proving and allows users to manipulate geometric components and proof scripts based on structures rather than strokes. The results from evaluation study show that our system is well perceived and users have high satisfaction with the accuracy of sketch recognition, the effectiveness of visual hints, and the efficiency of structure-based manipulation.

Multimodal error correction for continuous handwriting recognition in pen-based user interfaces

In this paper, we describe a multimodal error correction mechanism. It allows the user to correct errors in continuous handwriting recognition naturally by simultaneously using pen gesture and speech. A multimodal fusion algorithm is designed to enhance recognition accuracies of handwriting and speech through cross-modal influence. We have performed preliminary evaluation experiments and the results show that this multimodal mechanism can efficiently correct the errors in continuous handwriting recognition.

Structuring and manipulating hand-drawn concept maps

Concept maps are an important tool to knowledge organization, representation, and sharing. Most current concept map tools do not provide full support for hand-drawn concept map creation and manipulation, largely due to the lack of methods to recognize hand-drawn concept maps. We propose a structure recognition method. Our algorithm can extract node blocks and link blocks of a hand-drawn concept map by combining dynamic programming and graph partitioning and then build a concept-map structure by relating extracted nodes and links. We also introduce structure-based intelligent manipulation technique of hand-drawn concept maps. Evaluation shows that our method has high structure recognition accuracy in real time, and the intelligent manipulation technique is efficient and effective. Based on the technique, a note-taking tool ‘IdeaNote’ is developed. It not only supports natural note taking, but also supports efficient note editing. Besides research related to hand-drawn concept map, I will also introduce our work on other ink computing techniques and the systems we have developed.

Multimodal Chinese text entry with speech and keypad on mobile devices

Chinese text entry is challenging on mobile devices which rely on keypad input. Entering one character may require many key presses. This paper proposes a multimodal text entry technique for Chinese. In this method, Chinese user can enter Chinese text by simultaneously using the simplified phonemic input method named Jianpin with keypad and speech utterance. The key of the technique is a multimodal fusion algorithm, which synchronizes speech and keypad input and fuses redundant information from two modalities to get the best candidate. A preliminary evaluation shows that users appreciate this technique and it could reduce key presses and enhance the input efficiency.