Dana Hughes

Texture recognition and localization in amorphous robotic skin.

We present a soft robotic skin that can recognize and localize texture using a distributed set of sensors and computational elements that are inspired by the Pacinian corpuscle, the fast adapting, uniformly spaced mechanoreceptor with a wide receptive field, which is responsive to vibrations due to rubbing or slip on the skin. Tactile sensing and texture recognition is important for controlled manipulation of objects and navigating in ones environment. Yet, providing robotic systems or prosthetic devices with such capability at high density and bandwidth remains challenging. Each sensor node in the presented skin is created by collocating computational elements with individual microphones. These nodes are networked in a lattice and embedded in EcoFlex rubber, forming an amorphous medium. Unlike existing skins consisting of passive sensor arrays that feed into a central computer, our approach allows for detecting, conditioning and processing of tactile signals in-skin, facilitating the use of high-bandwidth signals, such as vibration, and allowing nodes to respond only to signals of interest. Communication between nodes allows the skin to localize the source of a stimulus, such as rubbing or slip, in a decentralized manner. Signal processing by individual nodes allows the skin to estimate the material touched. Combining these two capabilities, the skin is able to convert high-bandwidth, spatiotemporal information into low-bandwidth, event-driven information. Unlike taxel-based sensing arrays, this amorphous approach greatly reduces the computational load on the central robotic system. We describe the design, analysis, construction, instrumentation and programming of the robotic skin. We demonstrate that a 2.8 square meter skin with 10 sensing nodes is capable of localizing stimulus to within 2 centimeters, and that an individual sensing node can identify 15 textures with an accuracy of 71%. Finally, we discuss how such a skin could be used for full-body sensing in existing robots, augment existing sensing modalities, and how this material may be useful in robotic grasping tasks.

Detecting and Identifying Tactile Gestures using Deep Autoencoders, Geometric Moments and Gesture Level Features

While several sensing modalities and transduction approaches have been developed for tactile sensing in robotic skins, there has been much less work towards extracting features for or identifying high-level gestures performed on the skin. In this paper, we investigate using deep neural networks with hidden Markov models (DNN-HMMs), geometric moments and gesture level features to identify a set of gestures performed on robotic skins. We demonstrate that these features are useful for identifying gestures, and predict a set of gestures from a 14-class dataset with 56% accuracy, and a 7-class dataset with 71% accuracy.

A soft, amorphous skin that can sense and localize textures

We present a soft, amorphous skin that can sense and localize textures. The skin consists of a series of sensing and computing elements that are networked with their local neighbors and mimic the function of the Pacinian corpuscle in human skin. Each sensor node samples a vibration signal at 1 KHz, transforms the signal into the frequency domain, and classifies up to 15 textures using logistic regression. By measuring the power spectrum of the signal and comparing it with its local neighbors, computing elements can then collaboratively estimate the location of the stimulus. The resulting low-bandwidth information, consisting of the texture probability distribution and its location are then routed to a sink anywhere in the skin in a multi-hop fashion. We describe the design, manufacturing, classification, localization and networking algorithms and experimentally validate the proposed approach. In particular, we demonstrate texture classification with 71% accuracy and centimeter accuracy in localization over an area of approximately three square feet using ten networked sensor nodes.

SwitchBack: an on-body RF-based gesture input device

We present SwitchBack, a novel e-textile input device that can register multiple forms of input (tapping and bi-directional swiping) with minimal calibration. The technique is based on measuring the input impedance of a 7 cm microstrip short-circuit stub consisting of a strip of conductive fabric separated from a conductive fabric ground plane (also made of conductive fabric) by a layer of denim. The input impedance is calculated by measuring the stubs reflection coefficient using a simple RF reflectometer circuit, operating at 900MHz. The input impedance of the stub is affected by the dielectric properties of the surrounding material, and changes in a predictable manner when touched. We present the theoretical formulation, device and circuit design, and experimental results. Future work is also discussed.

Fractal Dimensions of Music and Automatic Playlist Generation: Similarity Search via MP3 Song Uploads

We present an automated approach to music search and playlist generation based on fractal dimensions of music. We compute 372 power-law metrics per song capturing statistical proportions of musical material. Using attribute selection and principal component analysis, we have reduced these metrics to approximately 45 independent features. These have been shown to capture important aspects of music aesthetics and similarity. Through an audio-to-MIDI transcription process, users may upload MP3 songs as search queries, in real time. This new development enables construction of music recommendation systems, which may work with previously unknown music. Unlike Pandora, last.fm, and Genius, such systems will analyze the actual music (potentially like the human ear), as opposed to harvesting information from humans (e.g., websites, user preferences, or musicologist recommendations). This approach combines time-frequency and spectral processing, information retrieval and audio analysis, and music classification. We present two on-line demos, using corpora from Magnatune and 7digital.

Distributed Convolutional Neural Networks for Human Activity Recognition in Wearable Robotics.

We investigate distributing convolutional neural networks (CNNs) for human activity recognition across computing nodes collocated with sensors at specific regions (body, arms and legs) on the wearer. We compare four CNN architectures. A distributed CNN is implemented on a network of Intel Edison nodes, demonstrating the capability of performing real-time classification. Two use a centralized, monolithic approach, and two are distributed across a number of computing nodes. While the accuracy of the distributed approaches are slightly worse than those of the monolithic CNNs, exploiting the hierarchy of the problem turns out to require much less memory — and therefore computation — than the monolithic CNNs, and only modest communication rates between nodes in the model, making the approach viable for a wide range of distributed systems ranging from wearable robots to multi-robot swarms.

Recognizing social touch gestures using recurrent and convolutional neural networks

Deep learning approaches have been used to perform classification in several applications with high-dimensional input data. In this paper, we investigate the potential for deep learning for classifying affective touch on robotic skin in a social setting. Three models are considered, a convolutional neural network, a convolutional-recurrent neural network and an autoencoder-recurrent neural network. These models are evaluated on two publicly available affective touch datasets, and compared with models built to classify the same datasets. The deep learning approaches provide a similar level of accuracy, and allows gestures to be predicted in real-time at a rate of 6 to 9 Hertz. The memory requirements of the models demonstrate that they can be implemented on small, inexpensive microcontrollers, demonstrating that classification can be performed in the skin itself by collocating computing elements with the sensor array.

Intelligent RF-Based Gesture Input Devices Implemented Using e-Textiles

We present an radio-frequency (RF)-based approach to gesture detection and recognition, using e-textile versions of common transmission lines used in microwave circuits. This approach allows for easy fabrication of input swatches that can detect a continuum of finger positions and similarly basic gestures, using a single measurement line. We demonstrate that the swatches can perform gesture detection when under thin layers of cloth or when weatherproofed, providing a high level of versatility not present with other types of approaches. Additionally, using small convolutional neural networks, low-level gestures can be identified with a high level of accuracy using a small, inexpensive microcontroller, allowing for an intelligent fabric that reports only gestures of interest, rather than a simple sensor requiring constant surveillance from an external computing device. The resulting e-textile smart composite has applications in controlling wearable devices by providing a simple, eyes-free mechanism to input simple gestures.

Monterey Mirror: an experiment in interactive music performance combining evolutionary computation and Zipf’s law

Abstract Monterey Mirror is an experiment in interactive music performance. It is engages a human (the performer) and a computer (the mirror) in a game of playing, listening, and exchanging musical ideas. The computer side involves an interactive stochastic music generator which incorporates Markov models, genetic algorithms, and power-law metrics. This approach combines the predictive power of Markov models with the innovative power of genetic algorithms, using power-law metrics for fitness evaluation. These power-law metrics have been developed and refined in a decade-long project, which explores music information retrieval based on Zipf’s law and related power laws. We describe the architecture of Monterey Mirror, which can generate musical responses based on aesthetic variations of user input. We also explore how such a system may be used as a musical meta-instrument/environment in avant-garde music composition and performance projects.