Nan-Wei Gong

Mime: compact, low power 3D gesture sensing for interaction with head mounted displays

We present Mime, a compact, low-power 3D sensor for unencumbered free-form, single-handed gestural interaction with head-mounted displays (HMDs). Mime introduces a real-time signal processing framework that combines a novel three-pixel time-of-flight (TOF) module with a standard RGB camera. The TOF module achieves accurate 3D hand localization and tracking, and it thus enables motion-controlled gestures. The joint processing of 3D information with RGB image data enables finer, shape-based gestural interaction. Our Mime hardware prototype achieves fast and precise 3D gestural control. Compared with state-of-the-art 3D sensors like TOF cameras, the Microsoft Kinect and the Leap Motion Controller, Mime offers several key advantages for mobile applications and HMD use cases: very small size, daylight insensitivity, and low power consumption. Mime is built using standard, low-cost optoelectronic components and promises to be an inexpensive technology that can either be a peripheral component or be embedded within the HMD unit. We demonstrate the utility of the Mime sensor for HMD interaction with a variety of application scenarios, including 3D spatial input using close-range gestures, gaming, on-the-move interaction, and operation in cluttered environments and in broad daylight conditions.

Building Functional Prototypes Using Conductive Inkjet Printing

The recently developed conductive inkjet printing process enables conductive circuits to be created quickly, cheaply, and easily using a consumer-grade inkjet printer. In its basic form, the technique supports a single layer of wiring on a flexible substrate. This can be a valuable tool for pervasive computing research because it allows simple electronic circuits and devices to be built and iterated quickly, in an analogous manner to the use of 3D printers for prototyping mechanical structures. It is possible to rapidly create touch- and proximity-sensitive surfaces, to cut and fold the printed conductive patterns, and to augment them with off-the-shelf electronic components and custom-made subcircuits. The authors present the possibilities enabled by conductive inkjet printing, bringing together their previously published results and presenting their latest insights and findings. They consider these printing and fabrication techniques as a suite of tools for researchers and practitioners who wish to fabricate a variety of functional device prototypes. They aim to enable others to understand the strengths, weaknesses, and applicability of conductive inkjet printing across a range of pervasive computing applications. This article is part of a special issue on printing and fabrication.

Wearable Sensing for Dynamic Management of Dense Ubiquitous Media

Most visions of ubiquitous computing anticipate a world permeated by a dense sampling of sensors, many of which will be capable of capturing, analyzing, and transmitting personally relevant and potentially privacy-sensitive media, such as video, audio, and identification information. This paper describes a set of sensor platforms that we have designed to experiment with personalization, interaction, and control in such dense media capture environments.

Dynamic privacy management in pervasive sensor networks

This paper describes the design and implementation of a dynamic privacy management system aimed at enabling tangible privacy control and feedback in a pervasive sensor network. Our work began with the development of a potentially invasive sensor network (with high resolution video, audio, and motion tracking capabilities) featuring different interactive applications that created incentive for accepting this network as an extension of peoples daily social space. A user study was then conducted to evaluate several privacy management approaches - an active badge system for both online and on-site control, on/off power switches for physically disabling the hardware, and touch screen input control. Results from a user study indicated that an active badge for on-site privacy control is the most preferable method among all provided options. We present a set of results that yield insight into the privacy/benefit tradeoff from various sensing capabilities in pervasive sensor networks and how privacy settings and user behavior relate in these environments.

Multimedia content creation using societal-scale ubiquitous camera networks and human-centric wearable sensing

We present a novel approach to the creation of user-generated, documentary video using a distributed network of sensor-enabled video cameras and wearable on-body sensor devices. The wearable sensors are used to identify the subjects in view of the camera system and label the captured video with real-time human-centric social and physical behavioral information. With these labels, massive amounts of continually recorded video can be browsed, searched, and automatically stitched into cohesive multimedia content. This system enables naturally occurring human behavior to drive and control a multimedia content creation system in order to create video output that is understandable, informative, and/or enjoyable to its human audience. The collected sensor data is further utilized to enhance the created multimedia content such as by using the data to edit and/or generate audio score, determine appropriate pacing of edits, and control the length and type of audio and video transitions directly from the content of the captured media. We present the design of the platform, the design of the multimedia content creation application, and the evaluated results from several live runs of the complete system.

Inkjet-printed conductive patterns for physical manipulation of audio signals

In this demo paper, we present the realization of a completely aesthetically driven conductive image as a multi-modal music controller. Combining two emerging technologies - rapid prototyping with an off-the-shelf inkjet printer using conductive ink and parametric graphic design, we are able to create an interactive surface that is thin, flat, and flexible. This sensate surface can be conformally wrapped around a simple curved surface, and unlike touch screens, can accommodate complex structures and shapes such as holes on a surface. We present the design and manufacturing flow and discuss the technology behind this multi-modal sensing design. Our work seeks to offer a new dimension of designing sonic interaction with graphic tools, playing and learning music from a visual perspective and performing with expressive physical manipulation.

Cutting Edge Vision: Metal Embedded Optics for Smart Knives

This video presents a novel technique for embedding optic fibers into a metal blade to sense objects that the knife is cutting. In particular, we present a design for a kitchen knife with fiber optics between the edge of the blade and the handle, with a skin-color sensor that overcomes the complex conditions in the kitchen. Hoping this design will lead to future work on minimizing cooking injuries, our handheld device also includes a simple finger-protection mechanism in the form of a retracting blade. We present our novel hardware design, an initial study of imaging capabilities, and a discussion of future directions.

Experiences and challenges in deploying potentially invasive sensor systems for dynamic media applications

This paper describes a series of projects that explore a set of dynamic media applications built upon a potentially invasive sensor system — the Ubiquitous Media Portal, featuring high-resolution video and audio capture with user ID/tracking capabilities that we installed throughout our facility. In addition to sensors, the portals provide a display and loudspeaker to locally display information or manifest phenomena from virtual worlds. During an eight-month long period, we implemented four different applications to explore acceptance by our buildingwide users. Our results provide insight into how different levels of information presentation and perceived user control can influence the user acceptance and engagement with such sensor platforms in ubiquitous deployments.

Conductive inkjet printed DIY music control surface

We developed a novel music control sensate surface, which enables retrofit integration between any musical instruments with a versatile, customizable, and essentially cost-effective user interface. Our project presents new opportunities in customizable, flexible interface design since, unlike just using a touch screen, it adapts very well to non-square or non-flat surfaces or surfaces with holes. Our design requires an interactive circuit that is made in a computer-aided design environment and printed from a conductive inkjet printer on a PET substrate. This method allows us to create a functional decoration on the controller surface, combining graphic design and music performance with expressive physical manipulation. We present an example of implementation on an electric ukulele and provide several design examples to demonstrate the versatile capabilities of this system.