Nicoletta Adamo-Villani

SMILE: an immersive learning game for deaf and hearing children

This paper describes the implementation and evaluation of the second iteration of SMILE (Science and Math in an Immersive Learning Environment), an immersive learning game that employs a fantasy 3D virtual environment to engage deaf and hearing children in math and science-based educational tasks. In this second iteration we (1) combine strategies used in commercial computer games with lessons from educational research on learning and motivation in order to increase the effectiveness of the application and the fun of its use, and (2) assess usability and appeal through a user study. To our knowledge, SMILE is the first bilingual immersive VLE (Virtual Learning Environment) for deaf and hearing students combining key elements of successful computer games, emotionally appealing graphics, and realistic real-time 3D signing, with goal-oriented, standards-based learning activities that are grounded in research on effective pedagogy.

The graph camera

A conventional pinhole camera captures only a small fraction of a 3-D scene due to occlusions. We introduce the graph camera, a non-pinhole with rays that circumvent occluders to create a single layer image that shows simultaneously several regions of interest in a 3-D scene. The graph camera image exhibits good continuity and little redundancy. The graph camera model is literally a graph of tens of planar pinhole cameras. A fast projection operation allows rendering in feed-forward fashion, at interactive rates, which provides support for dynamic scenes. The graph camera is an infrastructure level tool with many applications. We explore the graph camera benefits in the contexts of virtual 3-D scene exploration and summarization, and in the context of real-world 3-D scene visualization. The graph camera allows integrating multiple video feeds seamlessly, which enables monitoring complex real-world spaces with a single image.

An immersive virtual environment for learning sign language mathematics

In this paper we describe the development of a new immersive 3D learning environment to increase mathematical skills of deaf children. The application teaches mathematical concepts and ASL (American Sign Language) math terminology through user interaction with fantasy 3D virtual signers and environments. The program can be displayed in immersive devices and includes a gesture control system comprised of a pair of pinch gloves and a 6-degrees-of-freedom wrist tracker.

Sign Language for K-8 Mathematics by 3D Interactive Animation

We present a new highly interactive computer animation tool to increase the mathematical skills of deaf children. We aim at increasing the effectiveness of (hearing) parents in teaching arithmetic to their deaf children, and the opportunity of deaf children to learn arithmetic via interactive media. Using state-of-the-art computer animation technology we have designed and modeled three 3-dimensional virtual signers. The signs for mathematical terminology, performed by a non-hearing signer, have been captured with an electro-mechanical motion capture system and have been applied to the 3D characters. Macromedia Director MX Shockwave Studio has provided the platform for the design of interactive learning activities and for Web delivery. Smoothness of motion (in real time) and interactivity have been achieved via a new method of programmable blending of animation segments. Our program is the first attempt at using interactive 3D animation to improve the mathematical abilities of deaf pupils. Though our research follows the general trend of systems of 3D animated sign language (none of which has been geared specifically toward mathematics), we have made progress of a nature qualitatively different from previous work. Focusing on the need to represent the signs with clarity, realism, and emotional appeal for deaf children, we have invested our research efforts in developing a very natural, non-mechanical animated representation of the signs. The realization of this natural gesture language by interactive computer animation has been the main challenge that this research has met and solved.

Two Novel Technologies for Accessible Math and Science Education

We are using the results of the study to improve the design of both programs. We plan to repeat these evaluations several times as development of both programs progresses. Evaluation with kindergarten and elementary school deaf children and their teachers will be done in collaboration with the Indiana School for the Deaf and will start in the fall of 2009. We will report the results in a future article. A strong need exists for solutions that allow deaf users to communicate and interact in an environment free of prejudice, stigma, technological barrier, or other obstacles. The fact that all children were able to engage with and complete the tasks in both test systems is encouraging.

Sketching human character animations by composing sequences from large motion database

Quick creation of 3D character animations is an important task in game design, simulations, forensic animation, education, training, and more. We present a framework for creating 3D animated characters using a simple sketching interface coupled with a large, unannotated motion database that is used to find the appropriate motion sequences corresponding to the input sketches. Contrary to the previous work that deals with static sketches, our input sketches can be enhanced by motion and rotation curves that improve matching in the context of the existing animation sequences. Our framework uses animated sequences as the basic building blocks of the final animated scenes, and allows for various operations with them such as trimming, resampling, or connecting by use of blending and interpolation. A database of significant and unique poses, together with a two-pass search running on the GPU, allows for interactive matching even for large amounts of poses in a template database. The system provides intuitive interfaces, an immediate feedback, and poses very small requirements on the user. A user study showed that the system can be used by novice users with no animation experience or artistic talent, as well as by users with an animation background. Both groups were able to create animated scenes consisting of complex and varied actions in less than 20 minutes.

The MathSigner: an interactive learning tool for American sign language

We present a new highly interactive program to teach American sign language (ASL) for K-3 mathematics by 3D animation. We aim at increasing: (1) the opportunity for deaf children to learn arithmetic via interactive media; (2) the effectiveness of (hearing) parents in teaching arithmetic to their deaf children; (3) the effectiveness of ASL K-8 mathematics teachers. Using state-of-the-art 3D computer animation technology we have created a learning tool to be used for K-8 classroom and home learning, and for closed captioning in interactive media and Internet.

Automated finger spelling by highly realistic 3D animation

We present the design of a new 3D animation tool for self-teaching (signing and reading) finger spelling—the first basic component in learning any sign language. We have designed a highly realistic hand with natural animation of the finger motions. Smoothness of motion (in real time) is achieved via programmable blending of animation segments. The hand is utilised by a programmed that automatically converts text to finger spelling with controllable playback speed and playback views. The programme can be operated in two modes for two basic applications: learning to read (finger spelling) and learning to sign. For both modes, multi-sided views are provided. In addition, for the signing mode, a mirror view is provided for the common technique of using a mirror for feedback in practising finger spelling. An additional view revealing the joint structure of the hand allows the signer to practise the subjective view without having to guess the position of the fingers. This is the first example of highly realistic 3D animation that can be used practically to teach a basic aspect of sign language. The method is applicable more generally to sign language, and this finger-spelling application should be regarded as a first step toward the extension of highly realistic 3D animation to sign language in general. [ABSTRACT FROM AUTHOR]

Serious Sustainability Challenge Game to Promote Teaching and Learning of Building Sustainability

AbstractThis paper presents ongoing research aimed at improving building sustainability education through the use of serious games. With the growth of the population worldwide and the increased demand on resources, there has been a growing emphasis on sustainable buildings. As architects, engineers and construction professionals are educated for the future, the complex area of building sustainability must be included in the curriculum; however, the literature shows no clear consensus on teaching methods or curriculum design. Growing evidence suggests that computer games could be effective tools in construction education as they can provide students with both theoretical and applied knowledge. In this paper we describe the development and initial evaluation of a serious game for learning sustainable building design principles and practices. The game is aimed at undergraduate students enrolled in Civil Engineering, Architecture and Building Construction Management programs. Findings from a formative study w...

The MathSigner: An Interactive Learning Tool for American Sign Language K-3 Mathematics

We present a new highly interactive program to teach American Sign Language (ASL) for K-3 mathematics by 3D animation. We aim at increasing: (1) the opportunity for deaf children to learn arithmetic via interactive media; (2) the effectiveness of (hearing) parents in teaching arithmetic to their deaf children; (3) the effectiveness of ASL K-8 mathematics teachers. Using state-of-the-art 3D computer animation technology we have created a learning tool to be used for K-8 classroom and home learning, and for closed captioning in interactive media and internet.