Rachel Blagojevic

Signing on the tactile line: A multimodal system for teaching handwriting to blind children

We present McSig, a multimodal system for teaching blind children cursive handwriting so that they can create a personal signature. For blind people handwriting is very difficult to learn as it is a near-zero feedback activity that is needed only occasionally, yet in important situations; for example, to make an attractive and repeatable signature for legal contracts. McSig aids the teaching of signatures by translating digital ink from the teachers stylus gestures into three non-visual forms: (1) audio pan and pitch represents the x and y movement of the stylus; (2) kinaesthetic information is provided to the student through a force-feedback haptic pen that mimics the teachers stylus movement; and (3) a physical tactile line on the writing sheet is created by the haptic pen. McSig has been developed over two major iterations of design, usability testing and evaluation. The final step of the first iteration was a short evaluation with eight visually impaired children. The results suggested that McSig had the highest potential benefit for congenitally and totally blind children and also indicated some areas where McSig could be enhanced. The second prototype incorporated significant modifications to the system, improving the audio, tactile and force-feedback. We then ran a detailed, longitudinal evaluation over 14 weeks with three of the congenitally blind children to assess McSigs effectiveness in teaching the creation of signatures. Results demonstrated the effectiveness of McSig—they all made considerable progress in learning to create a recognizable signature. By the end of ten lessons, two of the children could form a complete, repeatable signature unaided, the third could do so with a little verbal prompting. Furthermore, during this project, we have learnt valuable lessons about providing consistent feedback between different communications channels (by manual interactions, haptic device, pen correction) that will be of interest to others developing multimodal systems.

The power of automatic feature selection: Rubine on steroids

Digital ink features drive recognition engines. Intuitively, we understand that particular features are of more value for some problems than others. Likewise, inclusion of poor features may be detrimental to recognition success. Many different ink features have been proposed for ink recognition, and most work well for the context that they are employed. However given a new problem it is not clear which of the already defined features will be most useful. We have assembled and categorized a comprehensive feature library and use this with attribute selection algorithms to choose the best features for a specified problem. To verify the effectiveness of this approach the selected features are used to train a Rubines recognizer. We show that a set of complementary features is most effective: poor features adversely affect recognition as do two or more aliases of good features. We have composed a variant of a Rubine recognizer for 3 different datasets and compared these with the Rubines original features, a variant on this InkRubine and

Technical Section: Using data mining for digital ink recognition: Dividing text and shapes in sketched diagrams

1. The results show that feature selection can significantly improve recognition rates with this simple algorithm thus verifying our hypothesis that the right combination of features for a problem is one key to recognition success.

Automatic evaluation of sketch recognizers

The low accuracy rates of text-shape dividers for digital ink diagrams are hindering their use in real world applications. While recognition of handwriting is well advanced and there have been many recognition approaches proposed for hand drawn sketches, there has been less attention on the division of text and drawing ink. Feature based recognition is a common approach for text-shape division. However, the choice of features and algorithms are critical to the success of the recognition. We propose the use of data mining techniques to build more accurate text-shape dividers. A comparative study is used to systematically identify the algorithms best suited for the specific problem. We have generated dividers using data mining with diagrams from three domains and a comprehensive ink feature library. The extensive evaluation on diagrams from six different domains has shown that our resulting dividers, using LADTree and LogitBoost, are significantly more accurate than three existing dividers.

A data collection tool for sketched diagrams

We present our toolkit to automatically evaluate recognition algorithms. There are few published comparative evaluations of sketch recognition algorithms and those that exist do not provide benchmarking or direct comparisons because standardised data and an evaluation platform is not available. By unifying data collection, labelling and evaluation in one tool, fair, flexible and comprehensive evaluations are possible. Currently we have 6 existing recognizers integrated into this tool. With our initial evaluations of these recognizers we have observed that the context from which training data is taken has an effect on recognition success rates. These results suggest that an evaluation platform such as this is a powerful adjunct for sketch recognition research.

Rata.SSR: data mining for pertinent stroke recognizers

Repositories of digital ink sketches would be invaluable for testing and evaluation of sketch recognition software. However, there is no existing tool for flexible data collection and management of digital ink data for building repositories of hand drawn diagrams. We present a tool for the efficient collection, management and analysis of ink data. A resultant dataset records each ink stroke accompanied by participant and diagram information, stroke labels and measurements of various stroke features. This tool enables the effective construction of a large database of sketches to aid the development of recognition techniques.

CapTUI: Geometric Drawing with Tangibles on a Capacitive Multi-touch Display

While many approaches to digital ink recognition have been proposed, most lack flexibility and adaptability to provide acceptable recognition rates across a variety of problem spaces. Time and expert knowledge are required to build accurate recognizers for a new domain. This project uses selected algorithms from a data mining toolkit and a large feature library, to compose a tailored software component (Rata.SSR) that enables single stroke recognizer generation from a few example diagrams. We evaluated Rata.SSR against four popular recognizers with three data sets (one of our own and two from other projects). The results show that it outperforms other recognizers on all tests except recognizer and data set pairs (e.g. PaleoSketch recognizer and PaleoSketch data set) -- in these cases the difference is small, and Rata is more flexible. We hence demonstrate a flexible and adaptable procedure for adopting existing techniques to quickly generate accurate recognizers without extensive knowledge of either AI or data mining.

Tangeo: geometric drawing with tangibles on an interactive table-top

We present CapTUI, an innovative drawing tool that detects tangible drawing instruments on a capacitive multi-touch tablet. There are three core components to the system: the tangible hardware, the recognizer used to identify the tangibles and the drawing software that works in tandem with the tangibles to provide intelligent visual drawing guides. Our recognizable tangible drawing instruments are a ruler, protractor and set square. Users employ these familiar physical instruments to construct digital ink drawings on a tablet in an intuitive and engaging manner. The visual drawing guides enhance the experience by offering the user helpful cues and functionalities to assist them to draw more accurately. A user evaluation comparing CapTUI to an application with passive tools showed that users significantly preferred CapTUI and found that the visual guides provide greater accuracy when drawing.

Supervised machine learning for grouping sketch diagram strokes

We introduce Tangeo, a drawing system that combines tangible drawing tools, such as rulers, protractors and set squares with a table-top environment. Geometric drawing on computers is often constrained to abstract widget tools and metaphoric, indirect input methods such as mouse and keyboard. Tangeo allows users to construct geometric drawings in a more direct manner by manipulating virtual data with familiar physical objects and drawing with a finger. User evaluations on Tangeo yielded a high rate of user satisfaction and indicated that the system is effective at enhancing geometric drawing.

Automated labeling of ink stroke data

Grouping of strokes into semantically meaningful diagram elements is a difficult problem. Yet such grouping is needed if truly natural sketching is to be supported in intelligent sketch tools. Using a machine learning approach, we propose a number of new paired-stroke features for grouping and evaluate the suitability of a range of algorithms. Our evaluation shows the new features and algorithms produce promising results that are statistically better than the existing machine learning grouper.