Hua-Tsung Chen

Human action recognition using star skeleton

This paper presents a HMM-based methodology for action recogni-tion using star skeleton as a representative descriptor of human posture. Star skeleton is a fast skeletonization technique by connecting from centroid of target object to contour extremes. To use star skeleton as feature for action recognition, we clearly define the fea-ture as a five-dimensional vector in star fashion because the head and four limbs are usually local extremes of human shape. In our proposed method, an action is composed of a series of star skeletons over time. Therefore, time-sequential images expressing human action are transformed into a feature vector sequence. Then the fea-ture vector sequence must be transformed into symbol sequence so that HMM can model the action. We design a posture codebook, which contains representative star skeletons of each action type and define a star distance to measure the similarity between feature vec-tors. Each feature vector of the sequence is matched against the codebook and is assigned to the symbol that is most similar. Conse-quently, the time-sequential images are converted to a symbol posture sequence. We use HMMs to model each action types to be recognized. In the training phase, the model parameters of the HMM of each category are optimized so as to best describe the training symbol sequences. For human action recognition, the model which best matches the observed symbol sequence is selected as the recog-nized category. We implement a system to automatically recognize ten different types of actions, and the system has been tested on real human action videos in two cases. One case is the classification of 100 video clips, each containing a single action type. A 98% recog-nition rate is obtained. The other case is a more realistic situation in which human takes a series of actions combined. An action-series recognition is achieved by referring a period of posture history using a sliding window scheme. The experimental results show promising performance.

Physics-based ball tracking and 3D trajectory reconstruction with applications to shooting location estimation in basketball video

The demand for computer-assisted game study in sports is growing dramatically. This paper presents a practical video analysis system to facilitate semantic content understanding. A physics-based algorithm is designed for ball tracking and 3D trajectory reconstruction in basketball videos and shooting location statistics can be obtained. The 2D-to-3D inference is intrinsically a challenging problem due to the loss of 3D information in projection to 2D frames. One significant contribution of the proposed system lies in the integrated scheme incorporating domain knowledge and physical characteristics of ball motion into object tracking to overcome the problem of 2D-to-3D inference. With the 2D trajectory extracted and the camera parameters calibrated, physical characteristics of ball motion are involved to reconstruct the 3D trajectories and estimate the shooting locations. Our experiments on broadcast basketball videos show promising results. We believe the proposed system will greatly assist intelligence collection and statistics analysis in basketball games.

A Trajectory-Based Ball Tracking Framework with Visual Enrichment for Broadcast Baseball Videos *

Pitching contents play the key role in the resultant victory or defeat in a baseball game. Utilizing the physical characteristic of ball motion, this paper presents a trajectory-based framework for automatic ball tracking and pitching evaluation in broadcast baseball videos. The task of ball detection and tracking in broadcast baseball videos is very challenging because in video frames, the noises may cause many ball-like objects, the ball size is small, and the ball may deform due to its high speed movement. To overcome these challenges, we first define a set of filters to prune most non-ball objects but retain the ball, even if it is deformed. In ball position prediction and trajectory extraction, we analyze the 2D distribution of ball candidates and exploit the characteristic that the ball trajectory presents in a near parabolic curve in video frames. Most of the non-qualified trajectories are pruned, which greatly improves the computational efficiency. The missed balls can also be recovered in the trajectory by applying the position prediction. The experiments of ball tracking on the testing sequences of JPB, MLB and CPBL captured from different TV channels show promising results. The ball tracking framework is able to extract the ball trajectory, superimposed on the video, and in near real-time provide visual enrichment before the next pitch coming up without specific cameras or equipments set up in the stadiums. It can also be utilized in strategy analysis and intelligence statistics for player training.

Ball tracking and 3D trajectory approximation with applications to tactics analysis from single-camera volleyball sequences

Providing computer-assisted tactics analysis in sports is a growing trend. This paper presents an automatic system for ball tracking and 3D trajectory approximation from single-camera volleyball sequences as well as demonstrates several applications to tactics analysis. Ball tracking in volleyball video has great complexity due to the high density of players on the court and the complicated overlapping of ball-player. The 2D-to-3D inference is intrinsically challenging due to the loss of 3D information in projection to 2D frames. To overcome these challenges, we propose a two-phase ball tracking algorithm in which we first detect ball candidates for each frame, and then use them to compute the ball trajectories. With the aid of camera calibration, we involve physical characteristics of ball motion to approximate the 3D ball trajectory from the 2D trajectory. The visualization of 3D trajectory and the applications to trajectory-based tactics analysis not only assist the coaches and players in game study but also make game watching a whole new experience. The experiments on international volleyball games show encouraging results. We believe that the proposed framework can be extended and applied to various kinds of sports games.

Pattern-Based Near-Duplicate Video Retrieval and Localization on Web-Scale Videos

With the exponential growth of web multimedia contents, the Internet is rife with near-duplicate videos, the video copies applied with visual/temporal transformations and/or post productions. Two critical issues, copyright infringement and search result redundancy, arise accordingly. To resolve these problems, this paper proposes a spatiotemporal pattern-based approach under the hierarchical filter-and-refine framework for efficient and effective near-duplicate video retrieval and localization . Firstly, non-near-duplicate videos are fast filtered out through a computationally efficient data structure, termed pattern -based index tree (PI-tree). Then, an m- pattern -based dynamic programming (mPDP) algorithm is designed to localize near-duplicate segments and to re-rank the videos retrieved. The influence of time shift misalignment can be alleviated by time-shift m-pattern similarity (TPS) measurement. Comprehensive experiments on the five datasets are conducted to verify the effectiveness, efficiency, robustness, and scalability of the proposed approach. Convincing results demonstrate that our proposed approach outperforms the state-of-the-art approaches in terms of mean average precision (MAP) and normalized detection cost rate (NDCR) on the testing datasets. Furthermore, the proposed approach can achieve high quality of near-duplicate video localization in terms of quality frames (QF) and mean F1.

Shot Classification of Basketball Videos and its Application in Shooting Position Extraction

In this paper, we propose a system that can automatically segment a basketball video into several clips on the basis of a GOP-based scene change detection method. The length of each clip and the number of dominant color pixels of each frame are used to classify shots into close-up view, medium view, and full court view. Full court view shots are chosen to do advanced analyses such as ball tracking and parameter extracting for the transformation from a 3D real-world court to a 2D image. After that, we map points in the 2D image to the corresponding coordinates in a real-world court by some physical properties of the 3D shooting trajectory, and compute the statistics of all shooting positions. Eventually we can obtain the information about the most possible shooting positions of a professional basketball team, which is useful for opponents to adopt appropriate defense tactics.

Physics-Based Ball Tracking in Volleyball Videos with its Applications to Set Type Recognition and Action Detection

Despite a lot of research efforts in sports video processing, little work was done in volleyball video analysis due to the high density of players on the court and the complicated overlapping of player-player or ball-player, which lead to great complexity of object tracking for advanced analysis. For ball detection and trajectory extraction in volleyball videos, this paper presents a physics-based scheme which utilizes the motion characteristics to extract ball trajectory from lots of moving objects. The experiments of ball tracking show encouraging results. Moreover, based on game-specific properties, the ball trajectory can be exploited to recognize set types for tactics inference and to detect basic actions in the volleyball game for close-up presentation.

Recognizing tactic patterns in broadcast basketball video using player trajectory

The explosive growth of the sports fandom inspires much research on manifold sports video analyses and applications. The audience, sports fans, and even professionals require more than traditional highlight extraction or semantic summarization. Computer-assisted sports tactic analysis is inevitably in urging demand. Recognizing tactic patterns in broadcast basketball video is a challenging task due to its complicated scenes, varied camera motion, frequently occlusions between players, etc. In basketball games, the action screen means that an offensive player perform a blocking move via standing beside or behind a defender for freeing a teammate to shoot, to receive a pass, or to drive in for scoring. In this paper, we propose a screen-strategy recognition system capable of detecting and classifying screen patterns in basketball video. The proposed system automatically detects the court lines for camera calibration, tracks players, and discriminates the offensive/defensive team. Player trajectories are calibrated to the real-world court model for screen pattern recognition. Our experiments on broadcast basketball videos show promising results. Furthermore, the extracted player trajectories and the recognized screen patterns visualized on a court model indeed assist the coach/players or the fans in comprehending the tactics executed in basketball games informatively and efficiently.

Pitch-by-Pitch Extraction from Single View Baseball Video Sequences

This paper presents a novel method for reducing a baseball video segment from one batter to next batter into a more compact pitch-by-pitch video by pitching ball trajectory detection. The pitch-by-pitch video shows the complete pitching and batting process and largely reduces the source video data, making pitching analysis of broadcast baseball sequences an easier task. The proposed method has been tested for several long sequences, and promising results are reported.

Daytime Preceding Vehicle Brake Light Detection Using Monocular Vision

Advanced vehicle safety is a recently emerging issue appealed from the explosive population of car owners. Increasing driver assistance systems have been developed for warning drivers of potential hazards by analyzing the surroundings with sensors and/or cameras. Issuing vehicle deceleration and potential collision, brake lights are particularly important warning signals, allowing of no neglect. In this paper, we propose a vision-based daytime brake light detection system using a driving video recorder, which tends to be widespread used. At daytime, visual features, motions, and appearances of vehicles are highly visible. However, brake lights, on the contrary, are hard to notice due to low contrast between the brakes lights and environments. Without the significant characteristic of light scattering as at night, the proposed system extracts preceding vehicles with taillight symmetry verification, and then integrates both luminance and radial symmetry features to detect brake lights. A detection refinement process using temporal information is also employed for miss recovery. Experiments are conducted on a test data set collected by front-mounted driving video recorders, and the results verify that the proposed system can effectively detect brake lights at daytime, showing its good feasibility in real-world environments.