Yingying Jiang

ShadowStory: creative and collaborative digital storytelling inspired by cultural heritage

With the fast economic growth and urbanization of many developing countries come concerns that their children now have fewer opportunities to express creativity and develop collaboration skills, or to experience their local cultural heritage. We propose to address these concerns by creating technologies inspired by traditional arts, and allowing children to create and collaborate through playing with them. ShadowStory is our first attempt in this direction, a digital storytelling system inspired by traditional Chinese shadow puppetry. We present the design and implementation of ShadowStory and a 7-day field trial in a primary school. Findings illustrated that ShadowStory promoted creativity, collaboration, and intimacy with traditional culture among children, as well as interleaved childrens digital and physical playing experience.

Intelligent understanding of handwritten geometry theorem proving

Computer-based geometry systems have been widely used for teaching and learning, but largely based on mouse-and-keyboard interaction, these systems usually require users to draw figures by following strict task structures defined by menus, buttons, and mouse and keyboard actions. Pen-based designs offer a more natural way to develop geometry theorem proofs with hand-drawn figures and scripts. This paper describes a pen-based geometry theorem proving system that can effectively recognize hand-drawn figures and hand-written proof scripts, and accurately establish the correspondence between geometric components and proof steps. Our system provides dynamic and intelligent visual assistance to help users understand the process of proving and allows users to manipulate geometric components and proof scripts based on structures rather than strokes. The results from evaluation study show that our system is well perceived and users have high satisfaction with the accuracy of sketch recognition, the effectiveness of visual hints, and the efficiency of structure-based manipulation.

Unistroke gestures on multi-touch interaction: supporting flexible touches with key stroke extraction

Gesture inputs on multi-touch tabletops usually involve multiple fingers (more than two) and casual touchdowns or liftoffs of fingers. This flexibility of touch gestures allows more natural user interaction, but also poses new challenges for accurate recognition of multi-touch gestures. To address these challenges, we propose a new approach to recognize flexible multi-touch stroke gestures on tabletops. Based on a user study on multi-touch unistroke gestures, we develop a gesture recognition method by extracting key strokes embedded in flexible multi-touch input. Our evaluation study result shows that this method can greatly improve the recognition accuracy of flexible multi-touch unistroke gestures on tabletops.

Understanding, Manipulating and Searching Hand-Drawn Concept Maps

Concept maps are an important tool to organize, represent, and share knowledge. Building a concept map involves creating text-based concepts and specifying their relationships with line-based links. Current concept map tools usually impose specific task structures for text and link construction, and may increase cognitive burden to generate and interact with concept maps. While pen-based devices (e.g., tablet PCs) offer users more freedom in drawing concept maps with a pen or stylus more naturally, the support for hand-drawn concept map creation and manipulation is still limited, largely due to the lack of methods to recognize the components and structures of hand-drawn concept maps. This article proposes a method to understand hand-drawn concept maps. Our algorithm can extract node blocks, or concept blocks, and link blocks of a hand-drawn concept map by combining dynamic programming and graph partitioning, recognize the text content of each concept node, and build a concept-map structure by relating concepts and links. We also design an algorithm for concept map retrieval based on hand-drawn queries. With our algorithms, we introduce structure-based intelligent manipulation techniques and ink-based retrieval techniques to support the management and modification of hand-drawn concept maps. Results from our evaluation study show high structure recognition accuracy in real time of our method, and good usability of intelligent manipulation and retrieval techniques.

Structuring and manipulating hand-drawn concept maps

Concept maps are an important tool to knowledge organization, representation, and sharing. Most current concept map tools do not provide full support for hand-drawn concept map creation and manipulation, largely due to the lack of methods to recognize hand-drawn concept maps. We propose a structure recognition method. Our algorithm can extract node blocks and link blocks of a hand-drawn concept map by combining dynamic programming and graph partitioning and then build a concept-map structure by relating extracted nodes and links. We also introduce structure-based intelligent manipulation technique of hand-drawn concept maps. Evaluation shows that our method has high structure recognition accuracy in real time, and the intelligent manipulation technique is efficient and effective. Based on the technique, a note-taking tool ‘IdeaNote’ is developed. It not only supports natural note taking, but also supports efficient note editing. Besides research related to hand-drawn concept map, I will also introduce our work on other ink computing techniques and the systems we have developed.

Multimodal Chinese text entry with speech and keypad on mobile devices

Chinese text entry is challenging on mobile devices which rely on keypad input. Entering one character may require many key presses. This paper proposes a multimodal text entry technique for Chinese. In this method, Chinese user can enter Chinese text by simultaneously using the simplified phonemic input method named Jianpin with keypad and speech utterance. The key of the technique is a multimodal fusion algorithm, which synchronizes speech and keypad input and fuses redundant information from two modalities to get the best candidate. A preliminary evaluation shows that users appreciate this technique and it could reduce key presses and enhance the input efficiency.

Thumb widgets: apply thumb-tracking to enhance capabilities of multi-touch on mobile devices

While multi-touch design allows natural and flexible figure gestures in interacting with mobile devices, the palm-size screen often limits its potential in application and arouses issues such as inefficient mode switch and incomplete gesture. To enrich the vocabulary of finger gestures in user interaction, we present Thumb Widgets, a mechanism that adopts thumb as assisted input channel to enhance the capabilities of multi-touch on touch-based mobile devices. Thumb Widgets expand the design space in gesture-based UIs and promote efficiency for tasks such as mode switching, parameter setting, etc. Also, Thumb Widgets enable incomplete gestures to be performed around the periphery of the device.

An exploration of pen tail gestures for interactions

In this paper, we performed an exploration on the design and evaluation of pen tail gesture, an interaction method that allows the use of pen tail movement to initiate interactions. Based on our interviews with some designers and researchers who regularly used pen-based tools, we conducted three experiments to establish baseline criteria to distinguish intentional pen tail gestures from incidental pen tail movements, and to understand the basic movement behaviors in pen tail gestures. We developed designs and recognition methods of pen gestures, and implemented three application prototypes based on them. Our research can inspire some new designs of pen-based tools and enrich the design repertoire of pen-based user interfaces.

Combining hand-drawn concept maps with RFID tags

As concept maps are useful for knowledge organization, representation and sharing, people used to draw concept maps by pen in their daily life. And some concept maps are closely related to physical world. In this paper, we propose a kind of context-aware hand-drawn concept map with RFID tags, which is especially useful for pen-based mobile devices. First, the structure of the hand-drawn concept map is extracted. And then users can assign RFID tags to concept nodes. With the RFID labeled concept nodes, concept maps are linked to the physical world, which can facilitate users understanding and retrieval of hand-drawn concept maps.