Mose Sakashita

Stimulated percussions: method to control human for learning music by using electrical muscle stimulation

In musical performances, it is important to produce rhythms correctly. However, when beginners play musical instruments, it can be difficult for them to understand rhythms using only visual and auditory rhythm information. To solve this problem, we propose the Stimulated Percussions (SP) system, which generates rhythms on a computer and transfers them to a users muscles. In this study, we control the users arms and legs using electrical muscle stimulation (EMS). We attach electrodes near certain arm and leg muscles, and provide stimulation in a manner that allows users to reproduce the correct movement when they play instruments. Our system enables a single player or multiple players to correctly reproduce generated rhythms. Experimental results show that our system is useful for beginners learning musical instruments, because it allows accurate rhythms to be mastered through bodily sensations.

You as a Puppet: Evaluation of Telepresence User Interface for Puppetry

We propose an immersive telepresence system for puppetry that transmits a human performers body and facial movements into a puppet with audiovisual feedback to the performer. The cameras carried in place of puppets eyes stream live video to the HMD worn by the performer, so that performers can see the images from the puppets eyes with their own eyes and have a visual understanding of the puppets ambience. In conventional methods to manipulate a puppet (a hand-puppet, a string-puppet, and a rod-puppet), there is a need to practice manipulating puppets, and there is difficulty carrying out interactions with the audience. Moreover, puppeteers must be positioned exactly where the puppet is. The proposed system addresses these issues by enabling a human performer to manipulate the puppet remotely using his or her body and facial movements. We conducted several user studies with both beginners and professional puppeteers. The results show that, unlike the conventional method, the proposed system facilitates the manipulation of puppets especially for beginners. Moreover, this system allows performers to enjoy puppetry and fascinate audiences.

Yadori: mask-type user interface for manipulation of puppets

We propose a system for animatronics storytelling that enables performers to manipulate puppets by wearing a mask-type device on their faces.

Stimulated percussions: techniques for controlling human as percussive musical instrument by using electrical muscle stimulation

We propose Stimulated Percussions, a new method designed to beat out rhythm by controlling human bodies with percussions using electrical muscle stimulation (EMS). Numerous studies in the fields of both science and art have been carried out to expand the possibilities for untrained people to be able to play musical instruments. Especially, in some studies, EMS capable of controlling the human body is applied to play a particular musical instrument. However, these studies have focused on movements of the human body for playing one specific instrument that requires movement of the whole body. Therefore, in this study, we used EMS to evaluate the body movements associated with different muscles. Moreover, we set up individual rhythms to these body movements and discussed the instrument that could be played by using these rhythms. We believe this design method would be able to expand the possibility for support to play a musical instrument.

Transformed Human Presence for Puppetry

We propose a system for transmitting a human performers body and facial movements to a puppet with audiovisual feedback to the performer. The system consists of a head-mounted display (HMD) that shows the performer the video recording of the puppets view, a microphone for voice capture, and photoreflectors for detecting the mouth movements of the human performer. In conventional puppetry, there is also the need for practice in the manipulation of the puppets to achieve good performance. The proposed telepresence system addresses these issues by enabling the human performer to manipulate the puppet through their own body and facial movements. The proposed system is expected to contribute to the development of new applications of puppetry and expand the interactivity of puppetry and the scope of entertainment.

Redesign of Cartesian Diver for Underwater Expression Combining Dynamic Fabrication with Non-contact Manipulation

In this study, we aim to combine dynamic fabrication with non-contact manipulation system applying the mechanism of Cartesian Diver. To achieve this, we propose the design method for underwater objects and non-contact manipulation technique using water pressure with PID control. We successfully designed and manipulate the object by our method. We discussed the principles and methods to create a digitally designed and fabricated the diver and to stabilize it in the middle of water.

Materialization of motions: tangible representation of dance movements for learning and archiving

We propose a method to learn and archive dance movements by fabricating tangible three-dimensional (3D) human forms. We analyze the pattern of the tempo and rhythm of a music piece and fabricate the shape of a 3D body based on the motions of the dance performer by using a 3D printer. For the implementation, we employ a depth camera to capture 3D information of a dance movement. Appropriate movements are extracted from the file at every constant tempo by analyzing the tempo of the music piece played while the dance is performed. The 3D printer enables tangible modeling of the dance movements.

Digital fabrication and manipulation method for underwater display and entertainment

Underwater expression is attractive. It seems like underwater objects are floating like anti-gravity scape by buoyancy and it is also impressive that bubbles rise while refracting the light. In this work, we aim to combine digital fabrication with interactive technology and expand underwater expression. To achieve this, we focused on a classic science experiment called the Cartesian Diver. Because of growing interest in the materialization of computer graphics, digital fabrication technologies have recently emerged as one of the most important application fields in real-world-oriented computer graphics. In particular, research on digital fabrication that gives dynamics properties is common. Spin-it [Bächer et al. 2014] presents design method for spinning objects by optimizing rotational dynamics properties. Some studies use non-contact manipulation. For example, ZeroN [Lee et al. 2011] controls the magnetic field to manipulate the object and uses it as a floating screen and input user interface(UI). Our work connects digital fabrication and non-contact manipulation that uses the space transmission power (water pressure) around the object (the diver). [Koike et al. 2016] proposes a design and manipulation method for the diver. In this work, we updated the method and investigate stability of PID control. Furthermore, we propose some applications.

Haptic marionette: wrist control technology combined with electrical muscle stimulation and hanger reflex

Many devices and systems that directly control a users hands have been proposed in previous studies. As a method for controlling a users wrist, Hanger Reflex and Electrical Muscle Stimulation is often used. We propose a method combined Electrical muscle stimulation and Hanger Reflex. We use Hanger Reflex to elicit the supination and pronation, and EMS to cause the flexion and extension. We believe that the proposed method of this study contributes to the exploration of new devices and applications on the fields of haptics, virtual and augmented reality, mobile and wearable interfaces.