Joseph C. Tsai

Video falsifying by motion interpolation and inpainting

We change the behavior of actors in a video. For instance, the outcome of a 100-meter race in the Olympic game can be falsified. We track objects and segment motions using a modified mean shift mechanism. The resulting video layers can be played in different speeds and at different reference points with respect to the original video. In order to obtain a smooth movement of target objects, a motion interpolation mechanism is proposed based on continuous stick figures (i.e., a video of human skeleton) and video inpainting. The video inpainting mechanism is performed in a quasi-3D space via guided 3D patch matching for filling. Interpolated target objects and background layers are fused by using graph cut. It is hard to tell whether a falsified video is the original. We demonstrate the original and the falsified videos in our website at http://www.mine.tku.edu.tw/video_demo/). The proposed technique can be used to create special effects in movie industry.

Video Motion Interpolation for Special Effect Applications

Video forgery, also referred as video falsifying, is a technique for generating fake videos by altering, combining, or creating new video contents. For instance, the outcome of a 100 m race in the olympic game is forged as an example in this paper. We track objects and segment motions using a modified mean shift mechanism. The resulting video layers can be played in different speeds and from different reference points with respect to the original video. In order to obtain a smooth movement of target objects, a motion interpolation mechanism is proposed based on reference stick figures (i.e., a structure of human skeleton) and a video inpainting mechanism. The video inpainting mechanism is performed in a quasi-3-D space via guided 3-D patch matching. Interpolated target objects and background layers are then fused. The objective is to create a forged video, which is almost indistinguishable from the original video. We demonstrate the original and the forged videos in our Web site at http://member.mine.tku.edu.tw/www/TSMC09/. Although video forgery may create moral or legal issues, which is beyond the scope of this paper, our intension is to create special effects in video editing applications.

Video Editing Using Motion Inpainting

In this paper, we demonstrate a new motion in painting technique to allow users to change the dynamic texture used in a video background for special effect production. For instance, the dynamic texture of fire, smoke, water, cloud, and others can be edited through a series of automatic algorithms. Motion estimations of global and local textures are used. Video blending techniques are used in conjunction with a color balancing technique. The editing procedure will search for suitable patches in irregular shape blocks, to reproduce a realistic dynamic background, such as large waterfall, fire scene, or smoky background. The technique is suitable for making science fiction movies. We demonstrate the original and the falsified videos in our website at http://163.13.127.36/www/AINA12. Although video falsifying may create a moral problem, our intension is to create special effects in movie industry.

Human Motion Analysis

We propose a novel motion analysis algorithm by using the mean-shift segmentation and motion estimation technique. Mean shift algorithm is frequently used to extract objects from video according to its efficiency and robustness of non-rigid object tracking. For diminishing the computational complexity in searching process, an efficient block matching algorithm: cross-diamond-hexagonal search algorithm was used. In the motion analysis procedure, the stick figure of object obtained by thinning process is treated as guidance to gather the statistics of motion information. The experimental results show that the proposed method provides precise description of the behavior of object in several video sequences

A coordinate transformation system based on the human feature information

In this paper, we propose a method to find feature in human object that used SURF algorithm, and use this information into 3D coordinate that use coordinate system transformation. In our method, first we use thinning algorithm to obtained skeleton of object, and find the endpoints in skeleton. In the second step, we try to use those endpoints to cluster skeleton, and the part number of cluster is six. Then, to cluster human object that use cluster skeleton result. Third, we use SURF algorithm to find the feature in each part in the cluster object image. In this step, we also use SAD method to ensure are correct of feature points that after SURF algorithm treatment. Finally we use the coordinate system transformation method. In this step, we use image coordinate system into world coordinate system, and show those result in our experiments result.

Face inpainting by feature guidance

Face image partially occluded or damaged can be repaired automatically. We proposed a new inpainting algorithm, based on patch guidance deduced from an existing face database, to recover the damaged portions. This newly proposed concept of guided inpainting method produces seamless faces which are hardly seen drawbacks. Examples of our results can be retrieved from http://member.mine.tku.edu.tw/www/ISCAS09/.

Motion inpainting and extrapolation for special effect production

Special effect production is an important technology for movie industry. Although sophisticated equipments can be used to produce scenery such as fire and smoke, usually, the approach is expensive and perhaps dangerous. Can sceneries be re-produced based on existing videos using digital technology? The answer is yes but it is difficult. In this video demonstration, we propose new mechanisms using motion inpainting technologies. Dynamic texture can be altered and reused under control. In addition, actors with repeated motion can be interpolated or extrapolated and incorporated into the falsified scenery. We demonstrate using our tool to produce realistic video narratives based on existing videos. Five sets of results as well as the source videos used are available at http://member.mine.tku.edu.tw/www/ACMMM09VideoDemo.

Motion extrapolation for video story planning

We create video scenes using existing videos. A panorama is generated from background video, with foreground objects removed by video inpainting technique. A video planning script is provided by the user on the panorama with accurate timing of actors. Actions of these actors are extensions of existing cyclic motions, such as walking, tracked and extrapolated using our motion analysis techniques. Although the types of video story generated are limited, however, it is possible to use the mechanism proposed to generate forgery videos. Interested readers should visit our tool demonstration and forgery videos at http://member.mine.tku.edu.tw/www/ACMMM08-VideoPlanning.

A real-time hand gesture recognition system for daily information retrieval from the internet

Nowadays people are used to get daily information from Internet such as weather condition, news and financial information, among others. Though, in order to receiving these daily information, users have to repeat same mouse and keyboard actions, inducing waste of time and inconvenience. In order to improve these situations, we propose in this paper a system design that can easily get daily information without mouse and keyboard actions and make peoples life more convenient and easier. In this proposed system, we have implemented an approach that provides daily information retrieved from Internet, where users can operate this system with his hands¡¦ movements. Once selected the function by hand gestures, the system will report action information to users by synthetized speech. In a typical family, since each member has different requirements and needs, the system utilizes face recognition to identify each user, bringing up personalized services to each user. In this paper, we use PCA method to recognize faces as also hand gestures, and then a number of hand gestures and system controls are acquired and stored into this system. Results from a set of experiments indicate that the proposed system in a family environment with small-scale of face recognition show good performance as also good result in hand gesture recognition.

Constructing interactive multi-view videos based on image-based rendering

In this paper, we use image-based rendering IBR to develop a scene rotation mechanism. We shot several images in the same scene and computed the angles between images. A video is then composed, allowing users to select viewing angles when the video is playing. We made three kinds of assumptions that may affect the resulting video, and proved our assumptions by a series of experiments. Finally, we use video of realistic scenario and produce interactive video by the proposed method. The contribution also includes techniques to compute geometric parameters of the scene from one or more images.