Maki Sugimoto

A Tracking Method for 2D Canvas in MR-Based Interactive Painting System

We have proposed a mixed reality based painting system. In this paper, we tackle a problem in our previous painting system that fully relies on magnetic sensor that is attached on the canvas, the users needed to detach and attach a sensor on the canvas during painting when they want to switch the canvas. Instead of that, in this paper, we aim to automatically detect the shape of the canvas for registration purpose. Using the shape or region detection method such as MSER (maximally stable extremal regions), we detect and track the shape on the canvas on the captured camera image. We then compute the camera pose for virtually overlay the painting result. Using the brush device, we can draw and paint freely on the tracked canvases. We show that using visual based tracking method, we can generate the equivalent result compared to the result of using the sensor.

3D Reconstruction of Hand Postures by Measuring Skin Deformation on Back Hand

In this research, we propose a method for reconstructing hand posture by measuring the deformation of the back of the hand with a wearable device. The deformation of skin on the back of the hand can be measured by using several photo-reflective sensors attached to a wearable device. In the learning phase, our method constructs a regression model by using the data on hand posture captured by a depth camera and data on the skin deformation of the back of the hand captured by several photoreflective sensors. In the estimation phase, by using this regression model, the posture of the hand is reconstructed from the data of the photo-reflective sensors in real-time. The posture of fingers can be estimated without hindering the natural movement of the fingers since the deformation of the back of the hand is measured without directly measuring the position of the fingers. This method can be used by users to manipulate information in a virtual environment with their fingers. We conducted an experiment to evaluate the accuracy of reconstructing hand posture with the proposed system. CCS Concepts •Human-centered computing → Interaction devices;

Facial Performance Capture by Embedded Photo Reflective Sensors on A Smart Eyewear.

Facial performance capture is used for animation production that projects a performers facial expression to a computer graphics model. Retro-reflective markers and cameras are widely used for the performance capture. To capture expressions, we need to place markers on the performers face and calibrate the intrinsic and extrinsic parameters of cameras in advance. However, the measurable space is limited to the calibrated area. In this paper, we propose a system to capture facial performance using a smart eyewear with photo reflective sensors and machine learning technique.