Takashi G. Sato

Image processing techniques for high speed camera-based free-field optical communication

Optical communication through light-emitting diodes (LEDs) and video cameras is rapidly gaining attention due to the increasing pervassiveness of those devices, and because of its potential data capacity. However, the communication quality is greatly compromised by the low resolution of the imaging sensor which produces blurred images of the LEDs at long distances. On the other hand, the images recorded at high frame rates possess particular features that can be used to improve the reception. This paper suggests image processing techniques to detect and decode the optical signals of an array of LEDs. For the case of blurred images, detection of the LEDs is reinforced by a k-means clustering algorithm based on distance measurements derived from the linear correlation among pixel intensities along the time dimension. Experiments with a prototype show that the proposed algorithms can improve the bit error rate (BER) of the decoded signal. Furthermore, partial implementation on a General Purpose Graphics Processing Unit (GPGPU) is also addressed, and processing times are demonstrated.

A playback system that synchronizes the musical phrases with listener's respiration phases

We propose a novel sound presentation system that utilizes a listeners respiration (breath) information. The system uses sound data, targets of the respiration phase bounded to the sound data and the listeners respiration trace. In replaying the sound, the system attempts to change the replay speed to minimize the difference between the target of respiration phase and the observed (listeners) respiration phase. Thus, a listener using the system has more chances to listen to a specific phrase with a specific respiration phase. In an experiment, chromatic scale movements were presented with different replay strategies to evaluate the system. Although the participant did not know about the control system, they reported a difference in their arousal feeling. The results indicate that music presentation that follows the listeners respiration will have significant effect on the impression of sound, suggesting that we may extend the possibility of sound presentation.

A High-Speed Camera-Based Approach to Massive Sound Sensing With Optical Wireless Acoustic Sensors

This paper introduces an optical wireless audio system, which takes advantage of the parallel transmission feature offered by the arrangement of LEDs and a high-speed video camera. The motivation for building such system is the limitation encountered when deploying huge arrays of sound sensors, as existent wired and RF wireless microphones are undermined by complexity, energy, and bandwidth issues. In contrast, the proposed prototype is able to simultaneously capture 120 audio channels with full bandwidth each. Although the scalability to more channels is currently constrained by the camera interface hardware, our numerical analysis suggests that the proposed algorithms can decode optical signals of over 12 500 audio channels in a single GPU card. We discuss the various challenges involved in its deployment from the optical, image, and audio signal processing standpoints, and develop theoretical and practical solutions to accomplish a full real-time system integration. We further present examples of typical applications such as acoustic imaging of sound fields by beamforming with microphone arrays.

Presenting changes in acoustic features synchronously to respiration alters the affective evaluation of sound.

Synchronization of respiration to cyclic auditory stimuli is a well-observed phenomenon and known to have an effect on affective evaluation of the presented sound. However, no studies have separated the effect of the change in respiratory movement itself and that when there is synchrony between respiration and sound. In this study, we used a system that can change the acoustic features synchronously with the respiration phase and directly investigated the effect the synchrony has on affective ratings without changing respiratory movements. An acoustic stimulation was presented where the sound intensity (SI) or fundamental frequency (F0) was modulated in response to the participants respiration phase. Affective evaluations of the acoustic stimuli were made by using the Self-Assessment Manikin (SAM). The experiments compared synchronous and asynchronous conditions. In the synchronous condition, SI (or F0) was increased with inhalation (decreased with exhalation) or decreased with inhalation (increased with exhalation). In the asynchronous condition, a sound identical to that presented in the synchronous condition was replayed. The participants evaluated sounds that were acoustically the same but where the temporal relationship differed between respiration and the acoustic features. In our results, significantly higher arousal ratings were observed when the change in SI and respiration (inhalation or exhalation) was synchronous and when the increase in F0 and inhalation was synchronous. This suggests that the synchronous phenomenon between respiration and auditory stimuli can play a critical role in affective evaluation.

Audience excitement reflected in respiratory phase synchronization

This paper investigates respiration activity data obtained simultaneously from more than ten persons while they were attending a live concert. Physiological activity in a group setting is attracting increased attention since it was suggested that it differs from data obtained individually with recorded stimuli. To overcome the difficulty in obtaining physiological data in a real situation, we developed a respiratory measurement system based on optical camera communication. We used the system to simultaneously measure respiration data from 15 participants who were in the audience of a live concert. Using the data, we showed that the respiratory phase distribution was distorted during some of the program suggesting a synchronization effect. We also found a program that was exciting for the participants and a program where the synchronization effects were almost identical, which suggests that a respiratory phase index can possibly be used to evaluate the quality of concerts.

Tactile Phantom Sensation for Coaching Respiration Timing

Respiration coaching is one of the key factors for radiation therapies. However, there are relatively few studies relating to respiration coaching, and most of them use audio or visual cues. In this paper, we show that a tactile phantom sensation moving continuously on the back can be used to adequately coach respiration timing. By using the tactile modality, the device rarely interferes with other communication channels used by therapists. The phantom sensation simplifies the mechanical structure. Several parameters were studied to obtain optimal performance when utilizing the phantom sensation. In a series of experiments, we determined the proper position and duty ratio for the actuators. To evaluate the device performance, we conducted an interference test (Kraepelin test), and the results suggest that the developed device interferes little with cognitive tasks. The experiments suggest that participants can easily understand stimulation on the back in terms of respiration guidance and properly follow changes in the cycle period with changes in respiration activity.

Increase in synchronization of respiration by repeatedly listening to the same piece of music

We found that participants tend to change their respiration at specific timing in listening to the same piece of music. Although the timing of concurrences differs between participants, such frequency increased within a participant as he or she gain experience on the piece. Our result supports that there is a strong relationship between respiration and music as studies had shown and besides, timing factor seems to be important as tempo for the relationship. This will help us design a new type of music interfaces, which should utilise physiological data.